2024-04-22T20:52:53.288 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T20:52:53.288 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668616 --name teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --verbose -- /home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668616/orig.config.yaml 2024-04-22T20:52:53.350 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T20:52:53.456 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668616 branch: main description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7668616' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 56e81df3aeb98b717efe2ab0537a8c60249f95f7 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 56e81df3aeb98b717efe2ab0537a8c60249f95f7 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 6458 sha1: 56e81df3aeb98b717efe2ab0537a8c60249f95f7 sleep_before_teardown: 0 subset: 39/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_8254cb5a247caa8e3d35e534a1fefe2362a0a4b8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 targets: smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKQ33A/qvrrcOnQzjEXQ8PcTGr3XAmFN+joD5xQyvTiVfgwdkCRP4bI4iOp3puBk+U6L6tiiuvFdqHSE6c7KULM= smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLEdZTsqxyIODkdPU5o/RoDkfhGnS6zC/8rrvvFEEf9pCxTb7miEbWkhEAhtM73ZDEJpI58+AgIA8uyZGWs7OIg= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_20:08:13 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-04-22T20:52:53.456 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_8254cb5a247caa8e3d35e534a1fefe2362a0a4b8/qa; will attempt to use it 2024-04-22T20:52:53.456 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_8254cb5a247caa8e3d35e534a1fefe2362a0a4b8/qa/tasks 2024-04-22T20:52:53.457 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T20:52:53.458 INFO:teuthology.task.internal:Checking packages... 2024-04-22T20:52:53.482 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '56e81df3aeb98b717efe2ab0537a8c60249f95f7' 2024-04-22T20:52:53.482 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T20:52:53.482 INFO:teuthology.packaging:ref: None 2024-04-22T20:52:53.482 INFO:teuthology.packaging:tag: None 2024-04-22T20:52:53.482 INFO:teuthology.packaging:branch: main 2024-04-22T20:52:53.483 INFO:teuthology.packaging:sha1: 56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-22T20:52:53.483 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-04-22T20:52:53.713 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3198.g56e81df3 2024-04-22T20:52:53.715 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T20:52:53.722 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T20:52:53.722 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T20:52:53.732 INFO:teuthology.task.internal:Saving configuration 2024-04-22T20:52:53.754 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T20:52:53.760 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T20:52:53.787 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668616', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 20:47:59.516069', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKQ33A/qvrrcOnQzjEXQ8PcTGr3XAmFN+joD5xQyvTiVfgwdkCRP4bI4iOp3puBk+U6L6tiiuvFdqHSE6c7KULM='} 2024-04-22T20:52:53.807 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668616', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 20:47:59.514619', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLEdZTsqxyIODkdPU5o/RoDkfhGnS6zC/8rrvvFEEf9pCxTb7miEbWkhEAhtM73ZDEJpI58+AgIA8uyZGWs7OIg='} 2024-04-22T20:52:53.807 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T20:52:53.814 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-22T20:52:53.814 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-22T20:52:53.814 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T20:52:53.879 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc2b135c0d0>, signals=[15]) 2024-04-22T20:52:53.879 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T20:52:53.886 INFO:teuthology.task.internal:Opening connections... 2024-04-22T20:52:53.886 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T20:52:53.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T20:52:53.969 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-04-22T20:52:53.970 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T20:52:54.050 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T20:52:54.057 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-04-22T20:52:54.076 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-04-22T20:52:54.076 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-04-22T20:52:54.134 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-04-22T20:52:54.134 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="9" 2024-04-22T20:52:54.134 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-04-22T20:52:54.134 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-04-22T20:52:54.134 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="9" 2024-04-22T20:52:54.135 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el9" 2024-04-22T20:52:54.135 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T20:52:54.135 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-04-22T20:52:54.135 INFO:teuthology.orchestra.run.smithi038.stdout:LOGO="fedora-logo-icon" 2024-04-22T20:52:54.135 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T20:52:54.135 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-04-22T20:52:54.135 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T20:52:54.135 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T20:52:54.135 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T20:52:54.136 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-04-22T20:52:54.156 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-04-22T20:52:54.174 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-04-22T20:52:54.174 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-04-22T20:52:54.231 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-04-22T20:52:54.231 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-04-22T20:52:54.231 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-04-22T20:52:54.231 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-04-22T20:52:54.231 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-04-22T20:52:54.231 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-04-22T20:52:54.231 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T20:52:54.231 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-04-22T20:52:54.231 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-04-22T20:52:54.232 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T20:52:54.232 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-04-22T20:52:54.232 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T20:52:54.232 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T20:52:54.232 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T20:52:54.232 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-04-22T20:52:54.255 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T20:52:54.265 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T20:52:54.271 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T20:52:54.271 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-04-22T20:52:54.273 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-04-22T20:52:54.289 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T20:52:54.295 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T20:52:54.296 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-04-22T20:52:54.331 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-04-22T20:52:54.356 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T20:52:54.411 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T20:52:54.426 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-22T20:52:54.426 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-22T20:52:54.426 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T20:52:54.427 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T20:52:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T20:52:54.442 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-04-22T20:52:54.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T20:52:54.444 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-04-22T20:52:54.498 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-437.el9.x86_64 2024-04-22T20:52:54.498 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-437.el9.x86_64 2024-04-22T20:52:54.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-04-22T20:52:54.501 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-437.el9.x86_64 2024-04-22T20:52:54.501 INFO:teuthology.task.kernel:Running kernel on smithi038: 5.14.0-437.el9.x86_64 2024-04-22T20:52:54.501 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2024-04-22T20:52:56.594 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-04-22T20:52:57.227 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 7.8 MB/s | 8.0 MB 00:01 2024-04-22T20:52:58.487 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-04-22T20:53:03.573 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.2 MB 00:01 2024-04-22T20:53:05.231 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-22T20:53:06.576 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-22T20:53:11.773 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-04-22T20:53:13.722 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T20:53:13.722 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T20:53:13.811 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:53:13.812 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-04-22T20:53:13.812 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:53:13.913 DEBUG:teuthology.orchestra.run.smithi038:> echo no | sudo yum reinstall kernel || true 2024-04-22T20:53:14.434 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 08:53:11 PM UTC. 2024-04-22T20:53:14.566 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:53:14.574 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:53:14.575 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-04-22T20:53:14.575 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:53:14.575 INFO:teuthology.orchestra.run.smithi038.stdout:Reinstalling: 2024-04-22T20:53:14.575 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T20:53:14.575 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:53:14.575 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:53:14.575 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:53:14.575 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:53:14.575 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 255 k 2024-04-22T20:53:14.575 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 0 2024-04-22T20:53:14.576 INFO:teuthology.orchestra.run.smithi038.stderr:Operation aborted. 2024-04-22T20:53:14.632 INFO:teuthology.orchestra.run.smithi038.stdout:Is this ok [y/N]: 2024-04-22T20:53:14.633 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum reinstall -y kernel || true 2024-04-22T20:53:15.147 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:04 ago on Mon 22 Apr 2024 08:53:11 PM UTC. 2024-04-22T20:53:15.267 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:53:15.268 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:53:15.268 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-04-22T20:53:15.268 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:53:15.268 INFO:teuthology.orchestra.run.smithi038.stdout:Reinstalling: 2024-04-22T20:53:15.268 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T20:53:15.269 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:53:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:53:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:53:15.269 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:53:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 255 k 2024-04-22T20:53:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 0 2024-04-22T20:53:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T20:53:15.559 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.1 MB/s | 255 kB 00:00 2024-04-22T20:53:15.560 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:53:15.560 INFO:teuthology.orchestra.run.smithi038.stdout:Total 877 kB/s | 255 kB 00:00 2024-04-22T20:53:15.578 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T20:53:15.751 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T20:53:15.752 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T20:53:15.757 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T20:53:15.757 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T20:53:15.889 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T20:53:15.968 INFO:teuthology.orchestra.run.smithi038.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T20:53:15.968 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T20:53:16.698 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T20:53:16.699 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T20:53:17.404 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T20:53:17.404 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:53:17.404 INFO:teuthology.orchestra.run.smithi038.stdout:Reinstalled: 2024-04-22T20:53:17.405 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T20:53:17.405 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:53:17.405 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:53:17.571 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T20:53:17.620 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T20:53:17.620 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T20:53:17.620 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T20:53:17.620 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-22T20:53:17.621 DEBUG:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T20:53:17.650 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2024-04-22T20:53:17.671 DEBUG:teuthology.parallel:result is None 2024-04-22T20:53:20.068 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 893 kB/s | 19 MB 00:21 2024-04-22T20:53:26.943 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 1.9 MB/s | 6.2 MB 00:03 2024-04-22T20:53:28.635 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-04-22T20:53:31.962 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 6.5 MB/s | 21 MB 00:03 2024-04-22T20:53:36.945 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-22T20:53:38.801 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T20:53:38.801 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T20:53:38.894 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:53:38.895 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-22T20:53:38.895 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:53:39.043 DEBUG:teuthology.orchestra.run.smithi098:> echo no | sudo yum reinstall kernel || true 2024-04-22T20:53:39.538 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 08:53:36 PM UTC. 2024-04-22T20:53:39.657 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:53:39.667 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:53:39.667 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-04-22T20:53:39.667 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:53:39.667 INFO:teuthology.orchestra.run.smithi098.stdout:Reinstalling: 2024-04-22T20:53:39.667 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T20:53:39.667 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:53:39.667 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:53:39.668 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:53:39.668 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:53:39.668 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 255 k 2024-04-22T20:53:39.668 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 0 2024-04-22T20:53:39.668 INFO:teuthology.orchestra.run.smithi098.stderr:Operation aborted. 2024-04-22T20:53:39.723 INFO:teuthology.orchestra.run.smithi098.stdout:Is this ok [y/N]: 2024-04-22T20:53:39.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum reinstall -y kernel || true 2024-04-22T20:53:40.216 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:04 ago on Mon 22 Apr 2024 08:53:36 PM UTC. 2024-04-22T20:53:40.332 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:53:40.333 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:53:40.333 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-04-22T20:53:40.333 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:53:40.333 INFO:teuthology.orchestra.run.smithi098.stdout:Reinstalling: 2024-04-22T20:53:40.333 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T20:53:40.333 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:53:40.333 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:53:40.333 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:53:40.333 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:53:40.333 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 255 k 2024-04-22T20:53:40.333 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 0 2024-04-22T20:53:40.334 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-22T20:53:40.572 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.4 MB/s | 255 kB 00:00 2024-04-22T20:53:40.572 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:53:40.572 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.0 MB/s | 255 kB 00:00 2024-04-22T20:53:40.594 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-22T20:53:40.785 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-22T20:53:40.785 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-22T20:53:40.790 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-22T20:53:40.790 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-22T20:53:40.921 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-22T20:53:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T20:53:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T20:53:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T20:53:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T20:53:42.337 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T20:53:42.337 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:53:42.337 INFO:teuthology.orchestra.run.smithi098.stdout:Reinstalled: 2024-04-22T20:53:42.337 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T20:53:42.337 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:53:42.337 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:53:42.509 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T20:53:42.556 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T20:53:42.556 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T20:53:42.556 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T20:53:42.556 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-22T20:53:42.556 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T20:53:42.583 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-04-22T20:53:42.604 DEBUG:teuthology.parallel:result is None 2024-04-22T20:53:42.604 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T20:53:42.612 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T20:53:42.612 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T20:53:42.615 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T20:53:42.659 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T20:53:42.699 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T20:53:42.745 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T20:53:42.745 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T20:53:42.748 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T20:53:42.781 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T20:53:42.788 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T20:53:42.789 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T20:53:42.819 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T20:53:42.851 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T20:53:42.856 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T20:53:42.860 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T20:53:42.867 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T20:53:42.869 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T20:53:42.877 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T20:53:42.877 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T20:53:42.912 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T20:53:42.937 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T20:53:42.947 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T20:53:42.947 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T20:53:42.980 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T20:53:42.996 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T20:53:43.078 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T20:53:43.130 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T20:53:43.214 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T20:53:43.256 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T20:53:43.256 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T20:53:43.324 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T20:53:43.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T20:53:43.416 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T20:53:43.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T20:53:43.537 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T20:53:43.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T20:53:43.600 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-04-22T20:53:43.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-04-22T20:53:43.640 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T20:53:43.678 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T20:53:44.171 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T20:53:44.179 INFO:teuthology.task.internal:Starting timer... 2024-04-22T20:53:44.179 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T20:53:44.189 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T20:53:44.197 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-22T20:53:44.198 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-04-22T20:53:44.264 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-04-22T20:53:44.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-04-22T20:53:44.316 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-04-22T20:53:44.318 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T20:53:44.318 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-04-22T20:53:44.338 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-04-22T20:53:44.338 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-04-22T20:53:44.362 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-04-22T20:53:44.362 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi098.front.sepia.ceph.com': 'permissive'} 2024-04-22T20:53:44.362 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T20:53:44.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T20:53:44.405 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T20:53:44.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T20:53:44.430 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T20:53:44.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-04-22T20:53:44.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-04-22T20:53:44.495 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T20:53:44.504 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T20:53:44.519 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T20:53:44.521 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com,smithi098.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T20:57:17.713 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi098.front.sepia.ceph.com')] 2024-04-22T20:57:17.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-22T20:57:17.715 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T20:57:17.813 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-04-22T20:57:17.906 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-22T20:57:17.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-22T20:57:17.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T20:57:17.982 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-04-22T20:57:18.061 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-22T20:57:18.062 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T20:57:18.073 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T20:57:18.074 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T20:57:18.075 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T20:57:18.077 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T20:57:18.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T20:57:18.107 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T20:57:18.121 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T20:57:18.132 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T20:57:18.145 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T20:57:18.146 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2024-04-22T20:57:18.157 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2024-04-22T20:57:18.168 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-04-22T20:57:18.172 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T20:57:18.179 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-04-22T20:57:18.186 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T20:57:18.191 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T20:57:18.205 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T20:57:18.241 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-04-22T20:57:18.244 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T20:57:18.244 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-04-22T20:57:18.244 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T20:57:18.244 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T20:57:18.245 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T20:57:18.245 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T20:57:18.262 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-04-22T20:57:18.265 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T20:57:18.265 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-04-22T20:57:18.265 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T20:57:18.265 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T20:57:18.265 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T20:57:18.265 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T20:57:18.267 INFO:teuthology.run_tasks:Running task install... 2024-04-22T20:57:18.275 DEBUG:teuthology.task.install:project ceph 2024-04-22T20:57:18.275 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '56e81df3aeb98b717efe2ab0537a8c60249f95f7'}} 2024-04-22T20:57:18.275 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '56e81df3aeb98b717efe2ab0537a8c60249f95f7'} 2024-04-22T20:57:18.275 INFO:teuthology.task.install:Using flavor: default 2024-04-22T20:57:18.281 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T20:57:18.281 INFO:teuthology.task.install:extra packages: [] 2024-04-22T20:57:18.281 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '56e81df3aeb98b717efe2ab0537a8c60249f95f7', 'tag': None, 'wait_for_package': False} 2024-04-22T20:57:18.282 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T20:57:18.282 INFO:teuthology.packaging:ref: None 2024-04-22T20:57:18.282 INFO:teuthology.packaging:tag: None 2024-04-22T20:57:18.282 INFO:teuthology.packaging:branch: quincy 2024-04-22T20:57:18.282 INFO:teuthology.packaging:sha1: 56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-22T20:57:18.282 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-04-22T20:57:18.284 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '56e81df3aeb98b717efe2ab0537a8c60249f95f7', 'tag': None, 'wait_for_package': False} 2024-04-22T20:57:18.284 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T20:57:18.284 INFO:teuthology.packaging:ref: None 2024-04-22T20:57:18.285 INFO:teuthology.packaging:tag: None 2024-04-22T20:57:18.285 INFO:teuthology.packaging:branch: quincy 2024-04-22T20:57:18.285 INFO:teuthology.packaging:sha1: 56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-22T20:57:18.285 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-04-22T20:57:18.511 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/70d0cb4b9d576828afb0a05dcf8732894d343ece/centos/9/flavors/default/ 2024-04-22T20:57:18.512 INFO:teuthology.task.install.rpm:Package version is 17.2.7-884.g70d0cb4b 2024-04-22T20:57:18.624 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/70d0cb4b9d576828afb0a05dcf8732894d343ece/centos/9/flavors/default/ 2024-04-22T20:57:18.624 INFO:teuthology.task.install.rpm:Package version is 17.2.7-884.g70d0cb4b 2024-04-22T20:57:18.629 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/70d0cb4b9d576828afb0a05dcf8732894d343ece/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/70d0cb4b9d576828afb0a05dcf8732894d343ece/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/70d0cb4b9d576828afb0a05dcf8732894d343ece/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T20:57:18.629 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T20:57:18.630 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T20:57:18.656 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T20:57:18.656 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T20:57:18.656 INFO:teuthology.packaging:ref: None 2024-04-22T20:57:18.656 INFO:teuthology.packaging:tag: None 2024-04-22T20:57:18.656 INFO:teuthology.packaging:branch: quincy 2024-04-22T20:57:18.656 INFO:teuthology.packaging:sha1: 56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-22T20:57:18.656 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T20:57:18.723 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T20:57:18.799 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T20:57:18.814 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/70d0cb4b9d576828afb0a05dcf8732894d343ece/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/70d0cb4b9d576828afb0a05dcf8732894d343ece/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/70d0cb4b9d576828afb0a05dcf8732894d343ece/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T20:57:18.814 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T20:57:18.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T20:57:18.841 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T20:57:18.841 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T20:57:18.841 INFO:teuthology.packaging:ref: None 2024-04-22T20:57:18.841 INFO:teuthology.packaging:tag: None 2024-04-22T20:57:18.841 INFO:teuthology.packaging:branch: quincy 2024-04-22T20:57:18.841 INFO:teuthology.packaging:sha1: 56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-22T20:57:18.841 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T20:57:18.863 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2024-04-22T20:57:18.865 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2024-04-22T20:57:18.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T20:57:18.980 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T20:57:19.044 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-04-22T20:57:19.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-04-22T20:57:19.100 INFO:teuthology.orchestra.run.smithi038.stdout:48 files removed 2024-04-22T20:57:19.127 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw 2024-04-22T20:57:19.269 INFO:teuthology.orchestra.run.smithi098.stdout:48 files removed 2024-04-22T20:57:19.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw 2024-04-22T20:57:19.741 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 226 kB/s | 77 kB 00:00 2024-04-22T20:57:19.950 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 194 kB/s | 77 kB 00:00 2024-04-22T20:57:20.117 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-04-22T20:57:20.309 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 34 kB/s | 11 kB 00:00 2024-04-22T20:57:20.475 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 5.5 kB/s | 1.8 kB 00:00 2024-04-22T20:57:20.718 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 4.8 kB/s | 1.8 kB 00:00 2024-04-22T20:57:21.126 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-22T20:57:21.410 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-22T20:57:24.936 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 7.7 MB/s | 19 MB 00:02 2024-04-22T20:57:30.243 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 2.4 MB/s | 19 MB 00:07 2024-04-22T20:57:30.912 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.2 MB 00:02 2024-04-22T20:57:32.546 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-22T20:57:35.522 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.2 MB 00:01 2024-04-22T20:57:37.012 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-04-22T20:57:37.230 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 4.6 MB/s | 21 MB 00:04 2024-04-22T20:57:38.408 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-22T20:57:42.214 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-22T20:57:43.664 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-04-22T20:57:44.346 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:57:44.348 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:57:44.348 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-22T20:57:44.348 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:57:44.348 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-22T20:57:44.349 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 14 M 2024-04-22T20:57:44.349 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-04-22T20:57:44.349 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 3.1 M 2024-04-22T20:57:44.349 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 2.9 M 2024-04-22T20:57:44.349 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-22T20:57:44.349 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 5.0 M 2024-04-22T20:57:44.349 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 22 M 2024-04-22T20:57:44.349 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 25 k 2024-04-22T20:57:44.349 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T20:57:44.349 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 638 k 2024-04-22T20:57:44.349 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T20:57:44.349 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T20:57:44.350 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T20:57:44.350 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 457 k 2024-04-22T20:57:44.350 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T20:57:44.350 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 4.9 M 2024-04-22T20:57:44.350 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T20:57:44.350 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T20:57:44.350 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 45 k 2024-04-22T20:57:44.350 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 118 k 2024-04-22T20:57:44.350 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 156 k 2024-04-22T20:57:44.350 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T20:57:44.350 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 320 k 2024-04-22T20:57:44.350 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 293 k 2024-04-22T20:57:44.350 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 99 k 2024-04-22T20:57:44.350 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T20:57:44.351 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:57:44.351 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:57:44.351 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:57:44.351 INFO:teuthology.orchestra.run.smithi098.stdout:Install 22 Packages 2024-04-22T20:57:44.351 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 2 Packages 2024-04-22T20:57:44.351 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:57:44.351 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 58 M 2024-04-22T20:57:44.351 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-22T20:57:45.098 INFO:teuthology.orchestra.run.smithi098.stdout:(1/24): ceph-base-17.2.7-884.g70d0cb4b.el7.x86_ 9.1 MB/s | 5.0 MB 00:00 2024-04-22T20:57:45.140 INFO:teuthology.orchestra.run.smithi098.stdout:(2/24): ceph-selinux-17.2.7-884.g70d0cb4b.el7.x 603 kB/s | 25 kB 00:00 2024-04-22T20:57:45.242 INFO:teuthology.orchestra.run.smithi098.stdout:(3/24): libcephfs2-17.2.7-884.g70d0cb4b.el7.x86 6.2 MB/s | 638 kB 00:00 2024-04-22T20:57:45.299 INFO:teuthology.orchestra.run.smithi098.stdout:(4/24): libradosstriper1-17.2.7-884.g70d0cb4b.e 7.7 MB/s | 457 kB 00:00 2024-04-22T20:57:45.666 INFO:teuthology.orchestra.run.smithi098.stdout:(5/24): librgw2-17.2.7-884.g70d0cb4b.el7.x86_64 13 MB/s | 4.9 MB 00:00 2024-04-22T20:57:45.716 INFO:teuthology.orchestra.run.smithi098.stdout:(6/24): python3-ceph-argparse-17.2.7-884.g70d0c 907 kB/s | 45 kB 00:00 2024-04-22T20:57:45.792 INFO:teuthology.orchestra.run.smithi098.stdout:(7/24): python3-ceph-common-17.2.7-884.g70d0cb4 1.5 MB/s | 118 kB 00:00 2024-04-22T20:57:45.859 INFO:teuthology.orchestra.run.smithi098.stdout:(8/24): python3-cephfs-17.2.7-884.g70d0cb4b.el7 2.3 MB/s | 156 kB 00:00 2024-04-22T20:57:45.867 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:57:45.869 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:57:45.869 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-04-22T20:57:45.870 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:57:45.870 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-22T20:57:45.870 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 14 M 2024-04-22T20:57:45.870 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2024-04-22T20:57:45.870 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 3.1 M 2024-04-22T20:57:45.870 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 2.9 M 2024-04-22T20:57:45.870 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-04-22T20:57:45.870 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 5.0 M 2024-04-22T20:57:45.870 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 22 M 2024-04-22T20:57:45.870 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 25 k 2024-04-22T20:57:45.870 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T20:57:45.870 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 638 k 2024-04-22T20:57:45.871 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T20:57:45.871 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T20:57:45.871 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T20:57:45.871 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 457 k 2024-04-22T20:57:45.871 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T20:57:45.871 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 4.9 M 2024-04-22T20:57:45.871 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T20:57:45.871 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T20:57:45.871 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 45 k 2024-04-22T20:57:45.871 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 118 k 2024-04-22T20:57:45.871 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 156 k 2024-04-22T20:57:45.871 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T20:57:45.871 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 320 k 2024-04-22T20:57:45.871 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 293 k 2024-04-22T20:57:45.872 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 99 k 2024-04-22T20:57:45.872 INFO:teuthology.orchestra.run.smithi038.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T20:57:45.872 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:57:45.872 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:57:45.872 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:57:45.872 INFO:teuthology.orchestra.run.smithi038.stdout:Install 22 Packages 2024-04-22T20:57:45.872 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 2 Packages 2024-04-22T20:57:45.872 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:57:45.873 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 58 M 2024-04-22T20:57:45.873 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T20:57:45.925 INFO:teuthology.orchestra.run.smithi098.stdout:(9/24): python3-rados-17.2.7-884.g70d0cb4b.el7. 4.7 MB/s | 320 kB 00:00 2024-04-22T20:57:45.992 INFO:teuthology.orchestra.run.smithi098.stdout:(10/24): python3-rbd-17.2.7-884.g70d0cb4b.el7.x 4.3 MB/s | 293 kB 00:00 2024-04-22T20:57:46.042 INFO:teuthology.orchestra.run.smithi098.stdout:(11/24): python3-rgw-17.2.7-884.g70d0cb4b.el7.x 1.9 MB/s | 99 kB 00:00 2024-04-22T20:57:46.235 INFO:teuthology.orchestra.run.smithi098.stdout:(12/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 835 kB/s | 160 kB 00:00 2024-04-22T20:57:46.385 INFO:teuthology.orchestra.run.smithi098.stdout:(13/24): ceph-radosgw-17.2.7-884.g70d0cb4b.el7. 7.7 MB/s | 14 MB 00:01 2024-04-22T20:57:46.418 INFO:teuthology.orchestra.run.smithi098.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 247 kB/s | 45 kB 00:00 2024-04-22T20:57:46.510 INFO:teuthology.orchestra.run.smithi098.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-04-22T20:57:46.569 INFO:teuthology.orchestra.run.smithi098.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 725 kB/s | 42 kB 00:00 2024-04-22T20:57:46.636 INFO:teuthology.orchestra.run.smithi098.stdout:(17/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.6 MB/s | 662 kB 00:00 2024-04-22T20:57:46.945 INFO:teuthology.orchestra.run.smithi098.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 157 kB/s | 49 kB 00:00 2024-04-22T20:57:47.045 INFO:teuthology.orchestra.run.smithi098.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 649 kB/s | 309 kB 00:00 2024-04-22T20:57:47.287 INFO:teuthology.orchestra.run.smithi098.stdout:(20/24): ceph-common-17.2.7-884.g70d0cb4b.el7.x 8.2 MB/s | 22 MB 00:02 2024-04-22T20:57:47.312 INFO:teuthology.orchestra.run.smithi098.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 183 kB/s | 67 kB 00:00 2024-04-22T20:57:47.354 INFO:teuthology.orchestra.run.smithi098.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 5.1 MB/s | 1.6 MB 00:00 2024-04-22T20:57:47.779 INFO:teuthology.orchestra.run.smithi098.stdout:(23/24): librbd1-17.2.7-884.g70d0cb4b.el7.x86_6 6.3 MB/s | 2.9 MB 00:00 2024-04-22T20:57:48.132 INFO:teuthology.orchestra.run.smithi038.stdout:(1/24): ceph-common-17.2.7-884.g70d0cb4b.el7.x8 13 MB/s | 22 MB 00:01 2024-04-22T20:57:48.182 INFO:teuthology.orchestra.run.smithi038.stdout:(2/24): ceph-selinux-17.2.7-884.g70d0cb4b.el7.x 506 kB/s | 25 kB 00:00 2024-04-22T20:57:48.213 INFO:teuthology.orchestra.run.smithi098.stdout:(24/24): librados2-17.2.7-884.g70d0cb4b.el7.x86 3.3 MB/s | 3.1 MB 00:00 2024-04-22T20:57:48.217 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:57:48.218 INFO:teuthology.orchestra.run.smithi098.stdout:Total 15 MB/s | 58 MB 00:03 2024-04-22T20:57:48.257 INFO:teuthology.orchestra.run.smithi038.stdout:(3/24): libcephfs2-17.2.7-884.g70d0cb4b.el7.x86 8.3 MB/s | 638 kB 00:00 2024-04-22T20:57:48.280 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-22T20:57:48.307 INFO:teuthology.orchestra.run.smithi038.stdout:(4/24): libradosstriper1-17.2.7-884.g70d0cb4b.e 9.0 MB/s | 457 kB 00:00 2024-04-22T20:57:48.313 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-22T20:57:48.313 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-22T20:57:48.599 INFO:teuthology.orchestra.run.smithi038.stdout:(5/24): librgw2-17.2.7-884.g70d0cb4b.el7.x86_64 17 MB/s | 4.9 MB 00:00 2024-04-22T20:57:48.633 INFO:teuthology.orchestra.run.smithi038.stdout:(6/24): python3-ceph-argparse-17.2.7-884.g70d0c 1.3 MB/s | 45 kB 00:00 2024-04-22T20:57:48.684 INFO:teuthology.orchestra.run.smithi038.stdout:(7/24): python3-ceph-common-17.2.7-884.g70d0cb4 2.3 MB/s | 118 kB 00:00 2024-04-22T20:57:48.734 INFO:teuthology.orchestra.run.smithi038.stdout:(8/24): python3-cephfs-17.2.7-884.g70d0cb4b.el7 3.1 MB/s | 156 kB 00:00 2024-04-22T20:57:48.748 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-22T20:57:48.749 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-22T20:57:48.792 INFO:teuthology.orchestra.run.smithi038.stdout:(9/24): python3-rados-17.2.7-884.g70d0cb4b.el7. 5.4 MB/s | 320 kB 00:00 2024-04-22T20:57:48.851 INFO:teuthology.orchestra.run.smithi038.stdout:(10/24): python3-rbd-17.2.7-884.g70d0cb4b.el7.x 4.9 MB/s | 293 kB 00:00 2024-04-22T20:57:48.928 INFO:teuthology.orchestra.run.smithi038.stdout:(11/24): python3-rgw-17.2.7-884.g70d0cb4b.el7.x 1.3 MB/s | 99 kB 00:00 2024-04-22T20:57:49.293 INFO:teuthology.orchestra.run.smithi038.stdout:(12/24): ceph-base-17.2.7-884.g70d0cb4b.el7.x86 1.7 MB/s | 5.0 MB 00:02 2024-04-22T20:57:49.485 INFO:teuthology.orchestra.run.smithi038.stdout:(13/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 288 kB/s | 160 kB 00:00 2024-04-22T20:57:49.760 INFO:teuthology.orchestra.run.smithi038.stdout:(14/24): ceph-radosgw-17.2.7-884.g70d0cb4b.el7. 4.2 MB/s | 14 MB 00:03 2024-04-22T20:57:49.788 INFO:teuthology.orchestra.run.smithi038.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 92 kB/s | 45 kB 00:00 2024-04-22T20:57:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-22T20:57:50.247 INFO:teuthology.orchestra.run.smithi038.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 92 kB/s | 42 kB 00:00 2024-04-22T20:57:50.288 INFO:teuthology.orchestra.run.smithi038.stdout:(17/24): librdkafka-1.6.1-102.el9.x86_64.rpm 825 kB/s | 662 kB 00:00 2024-04-22T20:57:50.396 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-04-22T20:57:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:17.2.7-884.g70d0cb4b.el7.x86_64 2/26 2024-04-22T20:57:50.531 INFO:teuthology.orchestra.run.smithi038.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 201 kB/s | 49 kB 00:00 2024-04-22T20:57:50.557 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:17.2.7-884.g70d0cb4b.el7.x86_64 2/26 2024-04-22T20:57:50.637 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:17.2.7-884.g70d0cb4b.el7.x86_64 3/26 2024-04-22T20:57:50.656 INFO:teuthology.orchestra.run.smithi038.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 755 kB/s | 309 kB 00:00 2024-04-22T20:57:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:17.2.7-884.g70d0cb4b.el7.x86_64 4/26 2024-04-22T20:57:50.681 INFO:teuthology.orchestra.run.smithi038.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 447 kB/s | 67 kB 00:00 2024-04-22T20:57:50.763 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:17.2.7-884.g70d0cb4b.el7.x86_64 4/26 2024-04-22T20:57:50.810 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-04-22T20:57:50.839 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-04-22T20:57:50.915 INFO:teuthology.orchestra.run.smithi038.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2024-04-22T20:57:51.024 INFO:teuthology.orchestra.run.smithi038.stdout:(22/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 232 kB/s | 292 kB 00:01 2024-04-22T20:57:51.199 INFO:teuthology.orchestra.run.smithi038.stdout:(23/24): librbd1-17.2.7-884.g70d0cb4b.el7.x86_6 10 MB/s | 2.9 MB 00:00 2024-04-22T20:57:51.272 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-04-22T20:57:51.304 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:17.2.7-884.g70d0cb4b.el7.x86_64 8/26 2024-04-22T20:57:51.358 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:17.2.7-884.g70d0cb4b.el7.x86_64 8/26 2024-04-22T20:57:51.387 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-04-22T20:57:51.430 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:17.2.7-884.g70d0cb4b.el7.x 10/26 2024-04-22T20:57:51.466 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:17.2.7-884.g70d0cb4b.el7.x86_64 11/26 2024-04-22T20:57:51.528 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:17.2.7-884.g70d0cb4b.el7.x86_64 12/26 2024-04-22T20:57:51.565 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:17.2.7-884.g70d0cb4b.el7.x86_64 13/26 2024-04-22T20:57:51.630 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:17.2.7-884.g70d0cb4b.el7.x86_64 13/26 2024-04-22T20:57:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-04-22T20:57:51.699 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-04-22T20:57:51.732 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-04-22T20:57:51.874 INFO:teuthology.orchestra.run.smithi038.stdout:(24/24): librados2-17.2.7-884.g70d0cb4b.el7.x86 2.6 MB/s | 3.1 MB 00:01 2024-04-22T20:57:51.876 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:57:51.877 INFO:teuthology.orchestra.run.smithi038.stdout:Total 9.6 MB/s | 58 MB 00:06 2024-04-22T20:57:51.939 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T20:57:51.978 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T20:57:51.978 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T20:57:52.016 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-04-22T20:57:52.053 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:17.2.7-884.g70d0cb4b.el7.x86_64 18/26 2024-04-22T20:57:52.142 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:17.2.7-884.g70d0cb4b.el7.x86_64 18/26 2024-04-22T20:57:52.216 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:17.2.7-884.g70d0cb4b.el7.x86_64 19/26 2024-04-22T20:57:52.247 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:17.2.7-884.g70d0cb4b.el7.x86 20/26 2024-04-22T20:57:52.437 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T20:57:52.437 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T20:57:53.692 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T20:57:54.035 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-04-22T20:57:54.166 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:17.2.7-884.g70d0cb4b.el7.x86_64 2/26 2024-04-22T20:57:54.245 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:17.2.7-884.g70d0cb4b.el7.x86_64 2/26 2024-04-22T20:57:54.325 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:17.2.7-884.g70d0cb4b.el7.x86_64 3/26 2024-04-22T20:57:54.362 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:17.2.7-884.g70d0cb4b.el7.x86_64 4/26 2024-04-22T20:57:54.450 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:17.2.7-884.g70d0cb4b.el7.x86_64 4/26 2024-04-22T20:57:54.496 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-04-22T20:57:54.525 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-04-22T20:57:54.725 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:17.2.7-884.g70d0cb4b.el7.x86_64 21/26 2024-04-22T20:57:54.769 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:17.2.7-884.g70d0cb4b.el7.x86_64 21/26 2024-04-22T20:57:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-04-22T20:57:55.006 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:17.2.7-884.g70d0cb4b.el7.x86_64 8/26 2024-04-22T20:57:55.069 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:17.2.7-884.g70d0cb4b.el7.x86_64 8/26 2024-04-22T20:57:55.106 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-04-22T20:57:55.150 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:17.2.7-884.g70d0cb4b.el7.x 10/26 2024-04-22T20:57:55.186 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:17.2.7-884.g70d0cb4b.el7.x86_64 11/26 2024-04-22T20:57:55.250 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:17.2.7-884.g70d0cb4b.el7.x86_64 12/26 2024-04-22T20:57:55.275 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:17.2.7-884.g70d0cb4b.el7.x86_64 21/26 2024-04-22T20:57:55.276 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:17.2.7-884.g70d0cb4b.el7.x86_64 13/26 2024-04-22T20:57:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:17.2.7-884.g70d0cb4b.el7.x86_64 22/26 2024-04-22T20:57:55.341 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:17.2.7-884.g70d0cb4b.el7.x86_64 13/26 2024-04-22T20:57:55.381 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-04-22T20:57:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:17.2.7-884.g70d0cb4b.el7.x86_64 22/26 2024-04-22T20:57:55.417 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T20:57:55.417 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T20:57:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:57:55.418 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-04-22T20:57:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:17.2.7-884.g70d0cb4b.el7.x86_64 23/26 2024-04-22T20:57:55.451 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-04-22T20:57:55.760 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-04-22T20:57:55.805 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:17.2.7-884.g70d0cb4b.el7.x86_64 18/26 2024-04-22T20:57:55.896 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:17.2.7-884.g70d0cb4b.el7.x86_64 18/26 2024-04-22T20:57:55.973 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:17.2.7-884.g70d0cb4b.el7.x86_64 19/26 2024-04-22T20:57:56.008 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:17.2.7-884.g70d0cb4b.el7.x86 20/26 2024-04-22T20:57:58.528 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:17.2.7-884.g70d0cb4b.el7.x86_64 21/26 2024-04-22T20:57:58.564 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:17.2.7-884.g70d0cb4b.el7.x86_64 21/26 2024-04-22T20:57:59.100 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:17.2.7-884.g70d0cb4b.el7.x86_64 21/26 2024-04-22T20:57:59.208 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:17.2.7-884.g70d0cb4b.el7.x86_64 22/26 2024-04-22T20:57:59.308 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:17.2.7-884.g70d0cb4b.el7.x86_64 22/26 2024-04-22T20:57:59.308 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T20:57:59.309 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T20:57:59.309 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:57:59.352 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:17.2.7-884.g70d0cb4b.el7.x86_64 23/26 2024-04-22T20:58:04.694 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:17.2.7-884.g70d0cb4b.el7.x86_64 23/26 2024-04-22T20:58:04.695 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-04-22T20:58:04.695 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-04-22T20:58:04.695 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-04-22T20:58:04.695 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-04-22T20:58:04.695 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-04-22T20:58:04.695 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-04-22T20:58:04.695 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-04-22T20:58:04.695 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:05.984 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:17.2.7-884.g70d0cb4b.el7.x86_64 24/26 2024-04-22T20:58:06.053 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-884.g70d0cb4b.el7.x86_64 24/26 2024-04-22T20:58:06.053 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T20:58:06.054 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T20:58:06.054 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T20:58:06.054 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T20:58:06.054 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:06.055 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-22T20:58:06.145 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-22T20:58:06.145 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-22T20:58:07.344 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-22T20:58:07.344 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/26 2024-04-22T20:58:07.344 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:17.2.7-884.g70d0cb4b.el7.x86_64 2/26 2024-04-22T20:58:07.344 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:17.2.7-884.g70d0cb4b.el7.x86_64 3/26 2024-04-22T20:58:07.344 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:17.2.7-884.g70d0cb4b.el7.x86_64 4/26 2024-04-22T20:58:07.344 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:17.2.7-884.g70d0cb4b.el7.x86_64 5/26 2024-04-22T20:58:07.345 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:17.2.7-884.g70d0cb4b.el7.x86_64 6/26 2024-04-22T20:58:07.345 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:17.2.7-884.g70d0cb4b.el7.x86_64 7/26 2024-04-22T20:58:07.345 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:17.2.7-884.g70d0cb4b.el7.x 8/26 2024-04-22T20:58:07.345 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:17.2.7-884.g70d0cb4b.el7.x86 9/26 2024-04-22T20:58:07.345 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:17.2.7-884.g70d0cb4b.el7.x86_64 10/26 2024-04-22T20:58:07.345 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:17.2.7-884.g70d0cb4b.el7.x86_64 11/26 2024-04-22T20:58:07.345 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:17.2.7-884.g70d0cb4b.el7.x86_64 12/26 2024-04-22T20:58:07.345 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:17.2.7-884.g70d0cb4b.el7.x86_64 13/26 2024-04-22T20:58:07.345 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-04-22T20:58:07.345 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-04-22T20:58:07.345 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-04-22T20:58:07.346 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-04-22T20:58:07.346 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-04-22T20:58:07.346 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-04-22T20:58:07.346 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-04-22T20:58:07.346 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-04-22T20:58:07.346 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-04-22T20:58:07.346 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:17.2.7-884.g70d0cb4b.el7.x86_64 23/26 2024-04-22T20:58:07.346 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-04-22T20:58:07.346 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:17.2.7-884.g70d0cb4b.el7.x86_64 25/26 2024-04-22T20:58:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-04-22T20:58:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:08.031 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-04-22T20:58:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:08.032 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-22T20:58:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T20:58:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T20:58:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T20:58:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T20:58:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T20:58:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T20:58:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T20:58:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T20:58:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T20:58:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:08.035 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:58:08.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-test 2024-04-22T20:58:08.913 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:26 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:58:09.039 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:58:09.040 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-22T20:58:09.040 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:09.040 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-22T20:58:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 48 M 2024-04-22T20:58:09.040 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-22T20:58:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 163 k 2024-04-22T20:58:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T20:58:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T20:58:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:09.041 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:58:09.041 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:09.041 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-04-22T20:58:09.041 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:09.041 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 48 M 2024-04-22T20:58:09.041 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 192 M 2024-04-22T20:58:09.042 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-22T20:58:09.340 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:17.2.7-884.g70d0cb4b.el7.x86_64 23/26 2024-04-22T20:58:09.340 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2024-04-22T20:58:09.340 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2024-04-22T20:58:09.340 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2024-04-22T20:58:09.341 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2024-04-22T20:58:09.341 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2024-04-22T20:58:09.341 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /root 2024-04-22T20:58:09.341 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2024-04-22T20:58:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:09.531 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): libcephsqlite-17.2.7-884.g70d0cb4b.el7.x 720 kB/s | 163 kB 00:00 2024-04-22T20:58:10.247 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 92 kB/s | 64 kB 00:00 2024-04-22T20:58:10.322 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): socat-1.7.4.1-5.el9.x86_64.rpm 300 kB/s | 305 kB 00:01 2024-04-22T20:58:10.655 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:17.2.7-884.g70d0cb4b.el7.x86_64 24/26 2024-04-22T20:58:10.734 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-884.g70d0cb4b.el7.x86_64 24/26 2024-04-22T20:58:10.734 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T20:58:10.734 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T20:58:10.734 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T20:58:10.734 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T20:58:10.734 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:10.736 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-22T20:58:10.816 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-22T20:58:10.816 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-22T20:58:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-22T20:58:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/26 2024-04-22T20:58:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:17.2.7-884.g70d0cb4b.el7.x86_64 2/26 2024-04-22T20:58:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:17.2.7-884.g70d0cb4b.el7.x86_64 3/26 2024-04-22T20:58:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:17.2.7-884.g70d0cb4b.el7.x86_64 4/26 2024-04-22T20:58:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:17.2.7-884.g70d0cb4b.el7.x86_64 5/26 2024-04-22T20:58:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:17.2.7-884.g70d0cb4b.el7.x86_64 6/26 2024-04-22T20:58:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:17.2.7-884.g70d0cb4b.el7.x86_64 7/26 2024-04-22T20:58:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:17.2.7-884.g70d0cb4b.el7.x 8/26 2024-04-22T20:58:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:17.2.7-884.g70d0cb4b.el7.x86 9/26 2024-04-22T20:58:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:17.2.7-884.g70d0cb4b.el7.x86_64 10/26 2024-04-22T20:58:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:17.2.7-884.g70d0cb4b.el7.x86_64 11/26 2024-04-22T20:58:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:17.2.7-884.g70d0cb4b.el7.x86_64 12/26 2024-04-22T20:58:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:17.2.7-884.g70d0cb4b.el7.x86_64 13/26 2024-04-22T20:58:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-04-22T20:58:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-04-22T20:58:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-04-22T20:58:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-04-22T20:58:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-04-22T20:58:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-04-22T20:58:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-04-22T20:58:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-04-22T20:58:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-04-22T20:58:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:17.2.7-884.g70d0cb4b.el7.x86_64 23/26 2024-04-22T20:58:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-04-22T20:58:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:17.2.7-884.g70d0cb4b.el7.x86_64 25/26 2024-04-22T20:58:12.810 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-04-22T20:58:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:12.811 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2024-04-22T20:58:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:12.811 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-22T20:58:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T20:58:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T20:58:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T20:58:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T20:58:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T20:58:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T20:58:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T20:58:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T20:58:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T20:58:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:12.813 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:58:12.833 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): ceph-test-17.2.7-884.g70d0cb4b.el7.x86_6 14 MB/s | 48 MB 00:03 2024-04-22T20:58:12.835 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:58:12.835 INFO:teuthology.orchestra.run.smithi098.stdout:Total 13 MB/s | 48 MB 00:03 2024-04-22T20:58:12.861 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-22T20:58:12.881 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-22T20:58:12.881 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-22T20:58:13.097 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-test 2024-04-22T20:58:13.242 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-22T20:58:13.242 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-22T20:58:13.644 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:30 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:58:13.751 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-22T20:58:13.767 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:58:13.768 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:58:13.768 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-04-22T20:58:13.768 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:58:13.768 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-22T20:58:13.768 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 48 M 2024-04-22T20:58:13.768 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-04-22T20:58:13.768 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 163 k 2024-04-22T20:58:13.769 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T20:58:13.769 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T20:58:13.769 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:13.769 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:58:13.769 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:58:13.769 INFO:teuthology.orchestra.run.smithi038.stdout:Install 4 Packages 2024-04-22T20:58:13.769 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:13.769 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 48 M 2024-04-22T20:58:13.769 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 192 M 2024-04-22T20:58:13.769 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T20:58:13.833 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-22T20:58:13.874 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-22T20:58:13.901 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:17.2.7-884.g70d0cb4b.el7.x86_64 3/4 2024-04-22T20:58:14.310 INFO:teuthology.orchestra.run.smithi038.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 305 kB 00:00 2024-04-22T20:58:14.369 INFO:teuthology.orchestra.run.smithi038.stdout:(2/4): libcephsqlite-17.2.7-884.g70d0cb4b.el7.x 509 kB/s | 163 kB 00:00 2024-04-22T20:58:14.420 INFO:teuthology.orchestra.run.smithi038.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 584 kB/s | 64 kB 00:00 2024-04-22T20:58:16.447 INFO:teuthology.orchestra.run.smithi038.stdout:(4/4): ceph-test-17.2.7-884.g70d0cb4b.el7.x86_6 20 MB/s | 48 MB 00:02 2024-04-22T20:58:16.451 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:58:16.451 INFO:teuthology.orchestra.run.smithi038.stdout:Total 18 MB/s | 48 MB 00:02 2024-04-22T20:58:16.478 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T20:58:16.499 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T20:58:16.499 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T20:58:16.882 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T20:58:16.883 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T20:58:17.419 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T20:58:17.508 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-22T20:58:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-22T20:58:17.574 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:17.2.7-884.g70d0cb4b.el7.x86_64 3/4 2024-04-22T20:58:18.262 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:17.2.7-884.g70d0cb4b.el7.x86_64 3/4 2024-04-22T20:58:18.340 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:17.2.7-884.g70d0cb4b.el7.x86_64 4/4 2024-04-22T20:58:20.535 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-test-2:17.2.7-884.g70d0cb4b.el7.x86_64 4/4 2024-04-22T20:58:20.536 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/4 2024-04-22T20:58:20.536 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:17.2.7-884.g70d0cb4b.el7.x86_64 2/4 2024-04-22T20:58:20.536 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-22T20:58:20.804 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-22T20:58:20.804 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:20.804 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-22T20:58:20.804 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:20.804 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:20.804 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T20:58:20.804 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T20:58:20.804 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:20.804 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:58:21.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph 2024-04-22T20:58:21.632 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:39 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:58:21.757 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:58:21.760 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-22T20:58:21.760 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:21.760 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-22T20:58:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 6.5 k 2024-04-22T20:58:21.760 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-22T20:58:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 2.1 M 2024-04-22T20:58:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 1.4 M 2024-04-22T20:58:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:17.2.7-884.g70d0cb4b.el7 ceph-noarch 240 k 2024-04-22T20:58:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 3.9 M 2024-04-22T20:58:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 15 M 2024-04-22T20:58:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T20:58:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T20:58:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T20:58:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T20:58:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T20:58:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T20:58:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T20:58:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T20:58:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T20:58:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T20:58:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T20:58:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T20:58:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T20:58:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T20:58:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T20:58:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T20:58:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T20:58:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T20:58:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T20:58:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T20:58:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T20:58:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T20:58:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T20:58:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T20:58:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T20:58:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T20:58:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T20:58:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:21.764 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:58:21.764 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:21.764 INFO:teuthology.orchestra.run.smithi098.stdout:Install 33 Packages 2024-04-22T20:58:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:21.764 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 27 M 2024-04-22T20:58:21.765 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 96 M 2024-04-22T20:58:21.765 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-22T20:58:22.181 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:17.2.7-884.g70d0cb4b.el7.x86_64 3/4 2024-04-22T20:58:22.343 INFO:teuthology.orchestra.run.smithi098.stdout:(1/33): ceph-17.2.7-884.g70d0cb4b.el7.x86_64.rp 37 kB/s | 6.5 kB 00:00 2024-04-22T20:58:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:17.2.7-884.g70d0cb4b.el7.x86_64 4/4 2024-04-22T20:58:22.526 INFO:teuthology.orchestra.run.smithi098.stdout:(2/33): ceph-mgr-17.2.7-884.g70d0cb4b.el7.x86_6 3.9 MB/s | 1.4 MB 00:00 2024-04-22T20:58:22.827 INFO:teuthology.orchestra.run.smithi098.stdout:(3/33): ceph-mon-17.2.7-884.g70d0cb4b.el7.x86_6 8.1 MB/s | 3.9 MB 00:00 2024-04-22T20:58:22.910 INFO:teuthology.orchestra.run.smithi098.stdout:(4/33): ceph-mgr-modules-core-17.2.7-884.g70d0c 2.8 MB/s | 240 kB 00:00 2024-04-22T20:58:22.952 INFO:teuthology.orchestra.run.smithi098.stdout:(5/33): ceph-mds-17.2.7-884.g70d0cb4b.el7.x86_6 2.6 MB/s | 2.1 MB 00:00 2024-04-22T20:58:23.052 INFO:teuthology.orchestra.run.smithi098.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-04-22T20:58:23.119 INFO:teuthology.orchestra.run.smithi098.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-22T20:58:23.161 INFO:teuthology.orchestra.run.smithi098.stdout:(8/33): python3-cryptography-36.0.1-4.el9.x86_6 5.9 MB/s | 1.2 MB 00:00 2024-04-22T20:58:23.212 INFO:teuthology.orchestra.run.smithi098.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 1.4 MB/s | 135 kB 00:00 2024-04-22T20:58:23.245 INFO:teuthology.orchestra.run.smithi098.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 1.5 MB/s | 125 kB 00:00 2024-04-22T20:58:23.295 INFO:teuthology.orchestra.run.smithi098.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-04-22T20:58:23.446 INFO:teuthology.orchestra.run.smithi098.stdout:(12/33): ceph-osd-17.2.7-884.g70d0cb4b.el7.x86_ 17 MB/s | 15 MB 00:00 2024-04-22T20:58:23.471 INFO:teuthology.orchestra.run.smithi098.stdout:(13/33): python3-mako-1.1.4-6.el9.noarch.rpm 763 kB/s | 172 kB 00:00 2024-04-22T20:58:23.496 INFO:teuthology.orchestra.run.smithi098.stdout:(14/33): python3-markupsafe-1.1.1-12.el9.x86_64 173 kB/s | 35 kB 00:00 2024-04-22T20:58:23.522 INFO:teuthology.orchestra.run.smithi098.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 677 kB/s | 51 kB 00:00 2024-04-22T20:58:23.547 INFO:teuthology.orchestra.run.smithi098.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 549 kB/s | 42 kB 00:00 2024-04-22T20:58:23.848 INFO:teuthology.orchestra.run.smithi098.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2024-04-22T20:58:23.948 INFO:teuthology.orchestra.run.smithi098.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-04-22T20:58:24.031 INFO:teuthology.orchestra.run.smithi098.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 337 kB/s | 172 kB 00:00 2024-04-22T20:58:24.048 INFO:teuthology.orchestra.run.smithi098.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-04-22T20:58:24.115 INFO:teuthology.orchestra.run.smithi098.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 278 kB/s | 23 kB 00:00 2024-04-22T20:58:24.132 INFO:teuthology.orchestra.run.smithi098.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 232 kB/s | 19 kB 00:00 2024-04-22T20:58:24.166 INFO:teuthology.orchestra.run.smithi098.stdout:(23/33): python3-cherrypy-18.6.1-2.el9.noarch.r 580 kB/s | 358 kB 00:00 2024-04-22T20:58:24.199 INFO:teuthology.orchestra.run.smithi098.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 234 kB/s | 20 kB 00:00 2024-04-22T20:58:24.233 INFO:teuthology.orchestra.run.smithi098.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 460 kB/s | 46 kB 00:00 2024-04-22T20:58:24.342 INFO:teuthology.orchestra.run.smithi098.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 449 kB/s | 79 kB 00:00 2024-04-22T20:58:24.375 INFO:teuthology.orchestra.run.smithi098.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 115 kB/s | 16 kB 00:00 2024-04-22T20:58:24.434 INFO:teuthology.orchestra.run.smithi098.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 979 kB/s | 90 kB 00:00 2024-04-22T20:58:24.506 INFO:teuthology.orchestra.run.smithi098.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 275 kB/s | 36 kB 00:00 2024-04-22T20:58:24.543 INFO:teuthology.orchestra.run.smithi098.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 794 kB/s | 272 kB 00:00 2024-04-22T20:58:24.550 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-test-2:17.2.7-884.g70d0cb4b.el7.x86_64 4/4 2024-04-22T20:58:24.550 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/4 2024-04-22T20:58:24.550 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:17.2.7-884.g70d0cb4b.el7.x86_64 2/4 2024-04-22T20:58:24.550 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-22T20:58:24.618 INFO:teuthology.orchestra.run.smithi098.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-04-22T20:58:24.643 INFO:teuthology.orchestra.run.smithi098.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-04-22T20:58:24.810 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-22T20:58:24.810 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:24.810 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-22T20:58:24.810 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:24.810 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:24.810 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T20:58:24.810 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T20:58:24.811 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:24.811 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:58:25.010 INFO:teuthology.orchestra.run.smithi098.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 847 kB/s | 427 kB 00:00 2024-04-22T20:58:25.013 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:58:25.013 INFO:teuthology.orchestra.run.smithi098.stdout:Total 8.3 MB/s | 27 MB 00:03 2024-04-22T20:58:25.071 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph 2024-04-22T20:58:25.130 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-22T20:58:25.166 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-22T20:58:25.167 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-22T20:58:25.432 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-22T20:58:25.433 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-22T20:58:25.604 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:42 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:58:25.732 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:58:25.734 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:58:25.734 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-04-22T20:58:25.734 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:58:25.734 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-22T20:58:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 6.5 k 2024-04-22T20:58:25.735 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-04-22T20:58:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 2.1 M 2024-04-22T20:58:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 1.4 M 2024-04-22T20:58:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:17.2.7-884.g70d0cb4b.el7 ceph-noarch 240 k 2024-04-22T20:58:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 3.9 M 2024-04-22T20:58:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 15 M 2024-04-22T20:58:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T20:58:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T20:58:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T20:58:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T20:58:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T20:58:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T20:58:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T20:58:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T20:58:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T20:58:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T20:58:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T20:58:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T20:58:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T20:58:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T20:58:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T20:58:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T20:58:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T20:58:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T20:58:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T20:58:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T20:58:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T20:58:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T20:58:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T20:58:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T20:58:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T20:58:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T20:58:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T20:58:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:25.737 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:58:25.737 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:58:25.737 INFO:teuthology.orchestra.run.smithi038.stdout:Install 33 Packages 2024-04-22T20:58:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:25.738 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 27 M 2024-04-22T20:58:25.738 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 96 M 2024-04-22T20:58:25.738 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T20:58:26.358 INFO:teuthology.orchestra.run.smithi038.stdout:(1/33): ceph-17.2.7-884.g70d0cb4b.el7.x86_64.rp 30 kB/s | 6.5 kB 00:00 2024-04-22T20:58:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-22T20:58:26.654 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-22T20:58:26.687 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-22T20:58:26.708 INFO:teuthology.orchestra.run.smithi038.stdout:(2/33): ceph-mds-17.2.7-884.g70d0cb4b.el7.x86_6 3.6 MB/s | 2.1 MB 00:00 2024-04-22T20:58:26.719 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-22T20:58:26.750 INFO:teuthology.orchestra.run.smithi038.stdout:(3/33): ceph-mgr-17.2.7-884.g70d0cb4b.el7.x86_6 2.3 MB/s | 1.4 MB 00:00 2024-04-22T20:58:26.811 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-22T20:58:26.842 INFO:teuthology.orchestra.run.smithi038.stdout:(4/33): ceph-mgr-modules-core-17.2.7-884.g70d0c 2.6 MB/s | 240 kB 00:00 2024-04-22T20:58:26.890 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-22T20:58:26.931 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-22T20:58:26.966 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-22T20:58:26.984 INFO:teuthology.orchestra.run.smithi038.stdout:(5/33): ceph-mon-17.2.7-884.g70d0cb4b.el7.x86_6 6.3 MB/s | 3.9 MB 00:00 2024-04-22T20:58:26.996 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-22T20:58:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-22T20:58:27.064 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-22T20:58:27.104 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-22T20:58:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-22T20:58:27.174 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-22T20:58:27.212 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-22T20:58:27.285 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-22T20:58:27.466 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-22T20:58:27.509 INFO:teuthology.orchestra.run.smithi038.stdout:(6/33): ceph-osd-17.2.7-884.g70d0cb4b.el7.x86_6 19 MB/s | 15 MB 00:00 2024-04-22T20:58:27.545 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-22T20:58:27.599 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-22T20:58:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-22T20:58:27.685 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-22T20:58:27.741 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-22T20:58:27.843 INFO:teuthology.orchestra.run.smithi038.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 253 kB/s | 253 kB 00:01 2024-04-22T20:58:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-22T20:58:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-22T20:58:28.062 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-22T20:58:28.157 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-22T20:58:28.216 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-22T20:58:28.331 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-22T20:58:28.359 INFO:teuthology.orchestra.run.smithi038.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 125 kB/s | 106 kB 00:00 2024-04-22T20:58:28.478 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-884.g70d0cb4b.el7.n 28/33 2024-04-22T20:58:28.501 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:17.2.7-884.g70d0cb4b.el7.x86_64 29/33 2024-04-22T20:58:28.527 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:17.2.7-884.g70d0cb4b.el7.x86_64 29/33 2024-04-22T20:58:28.527 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T20:58:28.527 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T20:58:28.527 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T20:58:28.527 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T20:58:28.527 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:28.726 INFO:teuthology.orchestra.run.smithi038.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 153 kB/s | 135 kB 00:00 2024-04-22T20:58:29.202 INFO:teuthology.orchestra.run.smithi038.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 148 kB/s | 125 kB 00:00 2024-04-22T20:58:29.344 INFO:teuthology.orchestra.run.smithi038.stdout:(11/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-04-22T20:58:29.402 INFO:teuthology.orchestra.run.smithi038.stdout:(12/33): python3-cryptography-36.0.1-4.el9.x86_ 519 kB/s | 1.2 MB 00:02 2024-04-22T20:58:29.428 INFO:teuthology.orchestra.run.smithi038.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 415 kB/s | 35 kB 00:00 2024-04-22T20:58:29.478 INFO:teuthology.orchestra.run.smithi038.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 678 kB/s | 51 kB 00:00 2024-04-22T20:58:29.503 INFO:teuthology.orchestra.run.smithi038.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 554 kB/s | 42 kB 00:00 2024-04-22T20:58:29.704 INFO:teuthology.orchestra.run.smithi038.stdout:(16/33): python3-urllib3-1.26.5-5.el9.noarch.rp 220 kB/s | 215 kB 00:00 2024-04-22T20:58:29.754 INFO:teuthology.orchestra.run.smithi038.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 158 kB/s | 43 kB 00:00 2024-04-22T20:58:29.821 INFO:teuthology.orchestra.run.smithi038.stdout:(18/33): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-04-22T20:58:29.835 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:17.2.7-884.g70d0cb4b.el7.x86_64 30/33 2024-04-22T20:58:29.846 INFO:teuthology.orchestra.run.smithi038.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 117 kB/s | 11 kB 00:00 2024-04-22T20:58:29.861 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:17.2.7-884.g70d0cb4b.el7.x86_64 30/33 2024-04-22T20:58:29.861 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T20:58:29.861 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T20:58:29.861 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T20:58:29.861 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T20:58:29.862 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:29.871 INFO:teuthology.orchestra.run.smithi038.stdout:(20/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 467 kB/s | 172 kB 00:00 2024-04-22T20:58:29.905 INFO:teuthology.orchestra.run.smithi038.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 302 kB/s | 18 kB 00:00 2024-04-22T20:58:29.947 INFO:teuthology.orchestra.run.smithi038.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-04-22T20:58:29.972 INFO:teuthology.orchestra.run.smithi038.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 193 kB/s | 19 kB 00:00 2024-04-22T20:58:29.997 INFO:teuthology.orchestra.run.smithi038.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 213 kB/s | 20 kB 00:00 2024-04-22T20:58:30.023 INFO:teuthology.orchestra.run.smithi038.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 615 kB/s | 46 kB 00:00 2024-04-22T20:58:30.048 INFO:teuthology.orchestra.run.smithi038.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-04-22T20:58:30.090 INFO:teuthology.orchestra.run.smithi038.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-04-22T20:58:30.132 INFO:teuthology.orchestra.run.smithi038.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 150 kB/s | 16 kB 00:00 2024-04-22T20:58:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:17.2.7-884.g70d0cb4b.el7.x86_64 31/33 2024-04-22T20:58:30.251 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:17.2.7-884.g70d0cb4b.el7.x86_64 31/33 2024-04-22T20:58:30.251 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T20:58:30.251 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T20:58:30.251 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T20:58:30.251 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T20:58:30.251 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:30.282 INFO:teuthology.orchestra.run.smithi038.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 384 kB/s | 90 kB 00:00 2024-04-22T20:58:30.316 INFO:teuthology.orchestra.run.smithi038.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 158 kB/s | 36 kB 00:00 2024-04-22T20:58:30.408 INFO:teuthology.orchestra.run.smithi038.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 833 kB/s | 230 kB 00:00 2024-04-22T20:58:30.441 INFO:teuthology.orchestra.run.smithi038.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-04-22T20:58:30.453 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:17.2.7-884.g70d0cb4b.el7.x86_64 32/33 2024-04-22T20:58:30.466 INFO:teuthology.orchestra.run.smithi038.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 133 kB/s | 20 kB 00:00 2024-04-22T20:58:30.472 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:58:30.472 INFO:teuthology.orchestra.run.smithi038.stdout:Total 5.7 MB/s | 27 MB 00:04 2024-04-22T20:58:30.476 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:17.2.7-884.g70d0cb4b.el7.x86_64 32/33 2024-04-22T20:58:30.476 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T20:58:30.477 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T20:58:30.477 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T20:58:30.477 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T20:58:30.477 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:30.541 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:17.2.7-884.g70d0cb4b.el7.x86_64 33/33 2024-04-22T20:58:30.595 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T20:58:30.636 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T20:58:30.636 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T20:58:30.921 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T20:58:30.922 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T20:58:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T20:58:32.181 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-2:17.2.7-884.g70d0cb4b.el7.x86_64 33/33 2024-04-22T20:58:32.181 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/33 2024-04-22T20:58:32.181 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:17.2.7-884.g70d0cb4b.el7.x86_64 2/33 2024-04-22T20:58:32.181 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:17.2.7-884.g70d0cb4b.el7.x86_64 3/33 2024-04-22T20:58:32.181 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:17.2.7-884.g70d0cb4b.el7.x86_64 4/33 2024-04-22T20:58:32.181 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:17.2.7-884.g70d0cb4b.el7.x86_64 5/33 2024-04-22T20:58:32.181 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-884.g70d0cb4b.el7.n 6/33 2024-04-22T20:58:32.181 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-22T20:58:32.181 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-22T20:58:32.181 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-22T20:58:32.181 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-22T20:58:32.181 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-22T20:58:32.182 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-22T20:58:32.182 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-22T20:58:32.182 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-22T20:58:32.182 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-22T20:58:32.182 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-22T20:58:32.182 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-22T20:58:32.182 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-22T20:58:32.182 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-22T20:58:32.182 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-22T20:58:32.182 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-22T20:58:32.182 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-22T20:58:32.182 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-22T20:58:32.182 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-22T20:58:32.183 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-22T20:58:32.184 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-22T20:58:32.184 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-22T20:58:32.184 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-22T20:58:32.184 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-22T20:58:32.184 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-22T20:58:32.184 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-22T20:58:32.184 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-22T20:58:32.186 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-22T20:58:32.219 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-22T20:58:32.252 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-22T20:58:32.357 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-22T20:58:32.433 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-22T20:58:32.489 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-22T20:58:32.524 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-22T20:58:32.553 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-22T20:58:32.587 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-22T20:58:32.621 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-22T20:58:32.662 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-22T20:58:32.705 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-22T20:58:32.739 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-22T20:58:32.769 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-22T20:58:32.847 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-22T20:58:33.000 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-22T20:58:33.115 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-22T20:58:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-22T20:58:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:33.129 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-22T20:58:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:17.2.7-884.g70d0cb4b.el7.noarch 2024-04-22T20:58:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T20:58:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T20:58:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T20:58:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T20:58:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T20:58:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T20:58:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T20:58:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T20:58:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T20:58:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T20:58:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T20:58:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T20:58:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T20:58:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T20:58:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T20:58:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T20:58:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T20:58:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T20:58:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T20:58:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T20:58:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T20:58:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T20:58:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T20:58:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T20:58:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T20:58:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T20:58:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T20:58:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:33.132 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:58:33.174 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-22T20:58:33.212 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-22T20:58:33.270 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-22T20:58:33.334 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-22T20:58:33.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-base 2024-04-22T20:58:33.533 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-22T20:58:33.608 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-22T20:58:33.689 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-22T20:58:33.788 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-22T20:58:33.823 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-22T20:58:33.950 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-22T20:58:33.952 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:51 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:58:34.030 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-base-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:58:34.070 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:58:34.071 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-22T20:58:34.071 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:58:34.109 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-884.g70d0cb4b.el7.n 28/33 2024-04-22T20:58:34.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install cephadm 2024-04-22T20:58:34.141 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:17.2.7-884.g70d0cb4b.el7.x86_64 29/33 2024-04-22T20:58:34.168 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:17.2.7-884.g70d0cb4b.el7.x86_64 29/33 2024-04-22T20:58:34.168 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T20:58:34.168 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T20:58:34.168 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T20:58:34.168 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T20:58:34.168 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:34.637 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:52 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:58:34.756 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:58:34.756 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:34.756 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-22T20:58:34.757 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:34.757 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-22T20:58:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:17.2.7-884.g70d0cb4b.el7 ceph-noarch 82 k 2024-04-22T20:58:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:34.757 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:58:34.757 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:34.757 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-04-22T20:58:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:34.757 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 82 k 2024-04-22T20:58:34.757 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 372 k 2024-04-22T20:58:34.757 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-22T20:58:35.121 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm-17.2.7-884.g70d0cb4b.el7.noarch.rpm 226 kB/s | 82 kB 00:00 2024-04-22T20:58:35.121 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:58:35.122 INFO:teuthology.orchestra.run.smithi098.stdout:Total 225 kB/s | 82 kB 00:00 2024-04-22T20:58:35.122 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-22T20:58:35.127 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-22T20:58:35.127 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-22T20:58:35.129 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-22T20:58:35.129 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-22T20:58:35.196 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-22T20:58:35.568 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:17.2.7-884.g70d0cb4b.el7.x86_64 30/33 2024-04-22T20:58:35.598 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:17.2.7-884.g70d0cb4b.el7.x86_64 30/33 2024-04-22T20:58:35.598 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T20:58:35.598 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T20:58:35.599 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T20:58:35.599 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T20:58:35.599 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:35.686 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 1/1 2024-04-22T20:58:35.757 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 1/1 2024-04-22T20:58:36.011 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:17.2.7-884.g70d0cb4b.el7.x86_64 31/33 2024-04-22T20:58:36.036 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:17.2.7-884.g70d0cb4b.el7.x86_64 31/33 2024-04-22T20:58:36.036 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T20:58:36.036 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T20:58:36.036 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T20:58:36.036 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T20:58:36.036 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:36.065 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 1/1 2024-04-22T20:58:36.245 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:17.2.7-884.g70d0cb4b.el7.x86_64 32/33 2024-04-22T20:58:36.269 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:17.2.7-884.g70d0cb4b.el7.x86_64 32/33 2024-04-22T20:58:36.270 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T20:58:36.270 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T20:58:36.270 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T20:58:36.270 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T20:58:36.270 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 1/1 2024-04-22T20:58:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:36.280 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-22T20:58:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 2024-04-22T20:58:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:36.280 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:58:36.336 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:17.2.7-884.g70d0cb4b.el7.x86_64 33/33 2024-04-22T20:58:36.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T20:58:37.011 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:55 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:58:37.133 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:58:37.133 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-04-22T20:58:37.134 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:37.134 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-22T20:58:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 137 k 2024-04-22T20:58:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:37.134 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:58:37.134 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:37.134 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-04-22T20:58:37.134 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:37.135 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 137 k 2024-04-22T20:58:37.135 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 379 k 2024-04-22T20:58:37.135 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-22T20:58:37.353 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-immutable-object-cache-17.2.7-884.g70d0cb4 634 kB/s | 137 kB 00:00 2024-04-22T20:58:37.353 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:58:37.354 INFO:teuthology.orchestra.run.smithi098.stdout:Total 628 kB/s | 137 kB 00:00 2024-04-22T20:58:37.354 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-22T20:58:37.361 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-22T20:58:37.361 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-22T20:58:37.416 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-22T20:58:37.416 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-22T20:58:37.557 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-22T20:58:37.620 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-884.g70d0cb4b.e 1/1 2024-04-22T20:58:37.644 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-884.g70d0cb4b.e 1/1 2024-04-22T20:58:37.645 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T20:58:37.645 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T20:58:37.645 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:38.018 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-2:17.2.7-884.g70d0cb4b.el7.x86_64 33/33 2024-04-22T20:58:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/33 2024-04-22T20:58:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:17.2.7-884.g70d0cb4b.el7.x86_64 2/33 2024-04-22T20:58:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:17.2.7-884.g70d0cb4b.el7.x86_64 3/33 2024-04-22T20:58:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:17.2.7-884.g70d0cb4b.el7.x86_64 4/33 2024-04-22T20:58:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:17.2.7-884.g70d0cb4b.el7.x86_64 5/33 2024-04-22T20:58:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-884.g70d0cb4b.el7.n 6/33 2024-04-22T20:58:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-22T20:58:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-22T20:58:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-22T20:58:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-22T20:58:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-22T20:58:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-22T20:58:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-22T20:58:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-22T20:58:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-22T20:58:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-22T20:58:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-22T20:58:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-22T20:58:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-22T20:58:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-22T20:58:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-22T20:58:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-22T20:58:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-22T20:58:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-22T20:58:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-22T20:58:38.021 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-22T20:58:38.021 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-22T20:58:38.021 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-22T20:58:38.022 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-22T20:58:38.022 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-22T20:58:38.022 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-22T20:58:38.022 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-22T20:58:38.470 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-884.g70d0cb4b.e 1/1 2024-04-22T20:58:38.471 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:38.471 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-22T20:58:38.471 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:38.471 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:38.471 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:58:38.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr 2024-04-22T20:58:38.939 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-22T20:58:38.939 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:38.940 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-22T20:58:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:17.2.7-884.g70d0cb4b.el7.noarch 2024-04-22T20:58:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T20:58:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T20:58:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T20:58:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T20:58:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T20:58:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T20:58:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T20:58:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T20:58:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T20:58:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T20:58:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T20:58:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T20:58:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T20:58:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T20:58:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T20:58:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T20:58:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T20:58:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T20:58:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T20:58:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T20:58:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T20:58:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T20:58:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T20:58:38.943 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T20:58:38.943 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T20:58:38.943 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T20:58:38.943 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T20:58:38.943 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:38.943 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:58:39.182 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:57 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:58:39.253 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-base 2024-04-22T20:58:39.259 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-mgr-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:58:39.302 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:58:39.303 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-22T20:58:39.303 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:58:39.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T20:58:39.782 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:56 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:58:39.863 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-base-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:58:39.868 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:57 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:58:39.906 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:58:39.907 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-04-22T20:58:39.907 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:58:39.978 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install cephadm 2024-04-22T20:58:39.988 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:58:39.989 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:39.989 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-22T20:58:39.990 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:39.990 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-22T20:58:39.990 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:17.2.7-884.g70d0cb4b.el7 ceph-noarch 3.4 M 2024-04-22T20:58:39.990 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-22T20:58:39.990 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:17.2.7-884.g70d0cb4b.el7 ceph-noarch 24 k 2024-04-22T20:58:39.990 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:17.2.7-884.g70d0cb4b.el7 ceph-noarch 15 k 2024-04-22T20:58:39.990 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-22T20:58:39.990 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T20:58:39.990 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T20:58:39.990 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-04-22T20:58:39.990 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-22T20:58:39.990 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:39.990 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:58:39.990 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:39.991 INFO:teuthology.orchestra.run.smithi098.stdout:Install 7 Packages 2024-04-22T20:58:39.991 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:39.991 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 3.7 M 2024-04-22T20:58:39.991 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 80 M 2024-04-22T20:58:39.991 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-22T20:58:40.266 INFO:teuthology.orchestra.run.smithi098.stdout:(1/7): ceph-prometheus-alerts-17.2.7-884.g70d0c 74 kB/s | 15 kB 00:00 2024-04-22T20:58:40.308 INFO:teuthology.orchestra.run.smithi098.stdout:(2/7): ceph-grafana-dashboards-17.2.7-884.g70d0 103 kB/s | 24 kB 00:00 2024-04-22T20:58:40.383 INFO:teuthology.orchestra.run.smithi098.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 77 kB/s | 9.0 kB 00:00 2024-04-22T20:58:40.408 INFO:teuthology.orchestra.run.smithi098.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 408 kB/s | 41 kB 00:00 2024-04-22T20:58:40.434 INFO:teuthology.orchestra.run.smithi098.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 611 kB/s | 31 kB 00:00 2024-04-22T20:58:40.502 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:57 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:58:40.592 INFO:teuthology.orchestra.run.smithi098.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.0 MB/s | 188 kB 00:00 2024-04-22T20:58:40.625 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:58:40.626 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:58:40.626 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-04-22T20:58:40.626 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:58:40.626 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-22T20:58:40.626 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:17.2.7-884.g70d0cb4b.el7 ceph-noarch 82 k 2024-04-22T20:58:40.626 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:40.626 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:58:40.626 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:58:40.626 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-04-22T20:58:40.626 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:40.627 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 82 k 2024-04-22T20:58:40.627 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 372 k 2024-04-22T20:58:40.627 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T20:58:40.917 INFO:teuthology.orchestra.run.smithi098.stdout:(7/7): ceph-mgr-dashboard-17.2.7-884.g70d0cb4b. 4.0 MB/s | 3.4 MB 00:00 2024-04-22T20:58:40.919 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:58:40.919 INFO:teuthology.orchestra.run.smithi098.stdout:Total 3.9 MB/s | 3.7 MB 00:00 2024-04-22T20:58:40.920 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-22T20:58:40.928 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-22T20:58:40.928 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-22T20:58:40.978 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm-17.2.7-884.g70d0cb4b.el7.noarch.rpm 233 kB/s | 82 kB 00:00 2024-04-22T20:58:40.979 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:58:40.979 INFO:teuthology.orchestra.run.smithi038.stdout:Total 232 kB/s | 82 kB 00:00 2024-04-22T20:58:40.979 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T20:58:40.985 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T20:58:40.985 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T20:58:40.987 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T20:58:40.987 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T20:58:40.990 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-22T20:58:40.990 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-22T20:58:41.068 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T20:58:41.295 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-22T20:58:41.360 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-22T20:58:41.394 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-22T20:58:41.441 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-22T20:58:41.481 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-22T20:58:41.500 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 1/1 2024-04-22T20:58:41.510 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-884.g70d0cb4b.el7.no 5/7 2024-04-22T20:58:41.601 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 1/1 2024-04-22T20:58:41.920 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 1/1 2024-04-22T20:58:42.138 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 1/1 2024-04-22T20:58:42.138 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:42.138 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-22T20:58:42.138 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 2024-04-22T20:58:42.139 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:42.139 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:58:42.346 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T20:58:42.509 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-884.g70d0cb4b.el7.n 6/7 2024-04-22T20:58:42.539 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-884.g70d0cb4b.el7.noarch 7/7 2024-04-22T20:58:42.879 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:59 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:58:43.003 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:58:43.003 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:58:43.004 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-04-22T20:58:43.004 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:58:43.004 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-22T20:58:43.004 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 137 k 2024-04-22T20:58:43.004 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:43.004 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:58:43.004 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:58:43.004 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-04-22T20:58:43.004 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:43.004 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 137 k 2024-04-22T20:58:43.004 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 379 k 2024-04-22T20:58:43.004 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T20:58:43.236 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-immutable-object-cache-17.2.7-884.g70d0cb4 594 kB/s | 137 kB 00:00 2024-04-22T20:58:43.237 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:58:43.237 INFO:teuthology.orchestra.run.smithi038.stdout:Total 590 kB/s | 137 kB 00:00 2024-04-22T20:58:43.237 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T20:58:43.246 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T20:58:43.246 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T20:58:43.306 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T20:58:43.306 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T20:58:43.451 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T20:58:43.546 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-884.g70d0cb4b.e 1/1 2024-04-22T20:58:43.571 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-884.g70d0cb4b.el7.noarch 7/7 2024-04-22T20:58:43.571 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-884.g70d0cb4b.el7.n 1/7 2024-04-22T20:58:43.571 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-884.g70d0cb4b.el7.noarch 2/7 2024-04-22T20:58:43.572 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-884.g70d0cb4b.el7.no 3/7 2024-04-22T20:58:43.572 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-22T20:58:43.572 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-22T20:58:43.572 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-22T20:58:43.572 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-884.g70d0cb4b.e 1/1 2024-04-22T20:58:43.572 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T20:58:43.572 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T20:58:43.572 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:43.821 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-22T20:58:43.821 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:43.821 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-22T20:58:43.821 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:17.2.7-884.g70d0cb4b.el7.noarch 2024-04-22T20:58:43.821 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:17.2.7-884.g70d0cb4b.el7.noarch 2024-04-22T20:58:43.821 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:17.2.7-884.g70d0cb4b.el7.noarch 2024-04-22T20:58:43.821 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-22T20:58:43.822 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-22T20:58:43.822 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T20:58:43.822 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T20:58:43.822 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:43.822 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:58:44.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T20:58:44.470 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-884.g70d0cb4b.e 1/1 2024-04-22T20:58:44.470 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:44.470 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-22T20:58:44.471 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:58:44.471 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:44.471 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:58:44.540 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:02 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:58:44.662 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:58:44.663 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-04-22T20:58:44.663 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-22T20:58:44.663 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-04-22T20:58:44.663 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-22T20:58:44.663 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-884.g70d0cb4b.el7 ceph-noarch 7.4 M 2024-04-22T20:58:44.663 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-22T20:58:44.663 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T20:58:44.663 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T20:58:44.663 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T20:58:44.663 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T20:58:44.663 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T20:58:44.663 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T20:58:44.663 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T20:58:44.664 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T20:58:44.664 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T20:58:44.664 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T20:58:44.664 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T20:58:44.664 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:44.664 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:58:44.664 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-04-22T20:58:44.664 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-04-22T20:58:44.664 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:44.665 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 38 M 2024-04-22T20:58:44.665 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 209 M 2024-04-22T20:58:44.665 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-22T20:58:44.672 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr 2024-04-22T20:58:45.142 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-04-22T20:58:45.201 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:02 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:58:45.209 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2024-04-22T20:58:45.251 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 301 kB/s | 33 kB 00:00 2024-04-22T20:58:45.283 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-mgr-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:58:45.301 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 363 kB/s | 18 kB 00:00 2024-04-22T20:58:45.327 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:58:45.328 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-04-22T20:58:45.328 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:58:45.360 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 603 kB/s | 35 kB 00:00 2024-04-22T20:58:45.398 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T20:58:45.461 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-04-22T20:58:45.553 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.6 MB/s | 244 kB 00:00 2024-04-22T20:58:45.778 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2024-04-22T20:58:45.895 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 250 kB 00:00 2024-04-22T20:58:45.950 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:02 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:58:45.962 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-04-22T20:58:46.076 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:58:46.076 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:58:46.077 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-04-22T20:58:46.077 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:58:46.077 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-22T20:58:46.077 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:17.2.7-884.g70d0cb4b.el7 ceph-noarch 3.4 M 2024-04-22T20:58:46.077 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-04-22T20:58:46.077 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:17.2.7-884.g70d0cb4b.el7 ceph-noarch 24 k 2024-04-22T20:58:46.077 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:17.2.7-884.g70d0cb4b.el7 ceph-noarch 15 k 2024-04-22T20:58:46.077 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-22T20:58:46.077 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T20:58:46.077 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T20:58:46.077 INFO:teuthology.orchestra.run.smithi038.stdout:Installing weak dependencies: 2024-04-22T20:58:46.077 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-22T20:58:46.077 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:46.078 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:58:46.078 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:58:46.078 INFO:teuthology.orchestra.run.smithi038.stdout:Install 7 Packages 2024-04-22T20:58:46.078 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:46.078 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 3.7 M 2024-04-22T20:58:46.078 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 80 M 2024-04-22T20:58:46.079 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T20:58:46.229 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-8 6.0 MB/s | 7.4 MB 00:01 2024-04-22T20:58:46.375 INFO:teuthology.orchestra.run.smithi038.stdout:(1/7): ceph-prometheus-alerts-17.2.7-884.g70d0c 72 kB/s | 15 kB 00:00 2024-04-22T20:58:46.400 INFO:teuthology.orchestra.run.smithi038.stdout:(2/7): ceph-grafana-dashboards-17.2.7-884.g70d0 106 kB/s | 24 kB 00:00 2024-04-22T20:58:46.600 INFO:teuthology.orchestra.run.smithi038.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 40 kB/s | 9.0 kB 00:00 2024-04-22T20:58:46.645 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-04-22T20:58:46.646 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:58:46.647 INFO:teuthology.orchestra.run.smithi098.stdout:Total 19 MB/s | 38 MB 00:01 2024-04-22T20:58:46.734 INFO:teuthology.orchestra.run.smithi038.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 123 kB/s | 41 kB 00:00 2024-04-22T20:58:46.767 INFO:teuthology.orchestra.run.smithi038.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 184 kB/s | 31 kB 00:00 2024-04-22T20:58:46.994 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-22T20:58:47.020 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-22T20:58:47.021 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-22T20:58:47.026 INFO:teuthology.orchestra.run.smithi038.stdout:(6/7): ceph-mgr-dashboard-17.2.7-884.g70d0cb4b. 3.9 MB/s | 3.4 MB 00:00 2024-04-22T20:58:47.051 INFO:teuthology.orchestra.run.smithi038.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 595 kB/s | 188 kB 00:00 2024-04-22T20:58:47.053 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:58:47.054 INFO:teuthology.orchestra.run.smithi038.stdout:Total 3.7 MB/s | 3.7 MB 00:00 2024-04-22T20:58:47.054 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T20:58:47.063 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T20:58:47.063 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T20:58:47.130 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T20:58:47.132 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T20:58:47.426 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-22T20:58:47.427 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-22T20:58:47.439 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T20:58:47.502 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-22T20:58:47.537 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-22T20:58:47.593 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-22T20:58:47.624 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-22T20:58:47.653 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-884.g70d0cb4b.el7.no 5/7 2024-04-22T20:58:48.185 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-22T20:58:48.250 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T20:58:48.294 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T20:58:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T20:58:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T20:58:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T20:58:48.652 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T20:58:48.705 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-884.g70d0cb4b.el7.n 6/7 2024-04-22T20:58:48.729 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T20:58:48.748 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-884.g70d0cb4b.el7.noarch 7/7 2024-04-22T20:58:49.370 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T20:58:49.482 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T20:58:49.829 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-884.g70d0cb4b.el7.noarch 7/7 2024-04-22T20:58:49.829 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-884.g70d0cb4b.el7.n 1/7 2024-04-22T20:58:49.829 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-884.g70d0cb4b.el7.noarch 2/7 2024-04-22T20:58:49.829 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-884.g70d0cb4b.el7.no 3/7 2024-04-22T20:58:49.829 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-22T20:58:49.829 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-22T20:58:49.829 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-22T20:58:50.087 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-22T20:58:50.087 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:50.087 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-22T20:58:50.088 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:17.2.7-884.g70d0cb4b.el7.noarch 2024-04-22T20:58:50.088 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:17.2.7-884.g70d0cb4b.el7.noarch 2024-04-22T20:58:50.088 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:17.2.7-884.g70d0cb4b.el7.noarch 2024-04-22T20:58:50.088 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-22T20:58:50.088 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-22T20:58:50.088 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T20:58:50.088 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T20:58:50.088 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:50.088 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:58:50.291 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T20:58:50.697 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T20:58:50.827 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:07 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:58:50.954 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:58:50.955 INFO:teuthology.orchestra.run.smithi038.stdout:====================================================================================== 2024-04-22T20:58:50.955 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-04-22T20:58:50.955 INFO:teuthology.orchestra.run.smithi038.stdout:====================================================================================== 2024-04-22T20:58:50.955 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-22T20:58:50.955 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-884.g70d0cb4b.el7 ceph-noarch 7.4 M 2024-04-22T20:58:50.955 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-04-22T20:58:50.955 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T20:58:50.955 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T20:58:50.956 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T20:58:50.956 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T20:58:50.956 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T20:58:50.956 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T20:58:50.956 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T20:58:50.956 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T20:58:50.956 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T20:58:50.956 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T20:58:50.956 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T20:58:50.956 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:50.956 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:58:50.956 INFO:teuthology.orchestra.run.smithi038.stdout:====================================================================================== 2024-04-22T20:58:50.956 INFO:teuthology.orchestra.run.smithi038.stdout:Install 12 Packages 2024-04-22T20:58:50.956 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:58:50.957 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 38 M 2024-04-22T20:58:50.957 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 209 M 2024-04-22T20:58:50.957 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T20:58:51.351 INFO:teuthology.orchestra.run.smithi038.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 938 kB/s | 195 kB 00:00 2024-04-22T20:58:51.502 INFO:teuthology.orchestra.run.smithi038.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 218 kB/s | 33 kB 00:00 2024-04-22T20:58:51.527 INFO:teuthology.orchestra.run.smithi038.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2024-04-22T20:58:51.644 INFO:teuthology.orchestra.run.smithi038.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 155 kB/s | 18 kB 00:00 2024-04-22T20:58:51.728 INFO:teuthology.orchestra.run.smithi038.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 423 kB/s | 35 kB 00:00 2024-04-22T20:58:51.740 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T20:58:51.769 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-884.g70d0cb 12/12 2024-04-22T20:58:52.070 INFO:teuthology.orchestra.run.smithi038.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-88 8.0 MB/s | 7.4 MB 00:00 2024-04-22T20:58:52.395 INFO:teuthology.orchestra.run.smithi038.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 751 kB/s | 244 kB 00:00 2024-04-22T20:58:52.504 INFO:teuthology.orchestra.run.smithi038.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.0 MB/s | 3.0 MB 00:01 2024-04-22T20:58:53.037 INFO:teuthology.orchestra.run.smithi038.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 470 kB/s | 250 kB 00:00 2024-04-22T20:58:53.729 INFO:teuthology.orchestra.run.smithi038.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 2.4 MB/s | 4.7 MB 00:02 2024-04-22T20:58:54.333 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-884.g70d0cb 12/12 2024-04-22T20:58:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-884.g70d0cb 1/12 2024-04-22T20:58:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T20:58:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T20:58:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T20:58:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T20:58:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T20:58:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T20:58:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T20:58:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T20:58:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T20:58:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T20:58:54.678 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T20:58:54.678 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:54.679 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-22T20:58:54.679 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:17.2.7-884.g70d0cb4b.el7.noarch 2024-04-22T20:58:54.679 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T20:58:54.679 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T20:58:54.679 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T20:58:54.679 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T20:58:54.679 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T20:58:54.679 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T20:58:54.679 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T20:58:54.679 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T20:58:54.679 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T20:58:54.679 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T20:58:54.679 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T20:58:54.680 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:54.680 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:58:55.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-rook 2024-04-22T20:58:55.556 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:13 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:58:55.679 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:58:55.680 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-22T20:58:55.680 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:55.681 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-22T20:58:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:17.2.7-884.g70d0cb4b.el7 ceph-noarch 47 k 2024-04-22T20:58:55.681 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-22T20:58:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T20:58:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T20:58:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T20:58:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T20:58:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T20:58:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T20:58:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T20:58:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T20:58:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T20:58:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T20:58:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T20:58:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T20:58:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:55.682 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:58:55.682 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:58:55.682 INFO:teuthology.orchestra.run.smithi098.stdout:Install 13 Packages 2024-04-22T20:58:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:58:55.683 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 2.2 M 2024-04-22T20:58:55.683 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 26 M 2024-04-22T20:58:55.683 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-22T20:58:55.988 INFO:teuthology.orchestra.run.smithi098.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 187 kB/s | 19 kB 00:00 2024-04-22T20:58:56.013 INFO:teuthology.orchestra.run.smithi098.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 201 kB/s | 26 kB 00:00 2024-04-22T20:58:56.080 INFO:teuthology.orchestra.run.smithi098.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-04-22T20:58:56.132 INFO:teuthology.orchestra.run.smithi098.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-04-22T20:58:56.138 INFO:teuthology.orchestra.run.smithi038.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.4 MB/s | 5.1 MB 00:03 2024-04-22T20:58:56.158 INFO:teuthology.orchestra.run.smithi098.stdout:(5/13): ceph-mgr-rook-17.2.7-884.g70d0cb4b.el7. 174 kB/s | 47 kB 00:00 2024-04-22T20:58:56.183 INFO:teuthology.orchestra.run.smithi098.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-04-22T20:58:56.208 INFO:teuthology.orchestra.run.smithi098.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.1 MB/s | 279 kB 00:00 2024-04-22T20:58:56.450 INFO:teuthology.orchestra.run.smithi098.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-04-22T20:58:56.476 INFO:teuthology.orchestra.run.smithi098.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 51 kB/s | 15 kB 00:00 2024-04-22T20:58:56.634 INFO:teuthology.orchestra.run.smithi098.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 476 kB/s | 203 kB 00:00 2024-04-22T20:58:56.660 INFO:teuthology.orchestra.run.smithi098.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-04-22T20:58:56.760 INFO:teuthology.orchestra.run.smithi098.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 716 kB/s | 90 kB 00:00 2024-04-22T20:58:56.793 INFO:teuthology.orchestra.run.smithi098.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2024-04-22T20:58:56.796 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:58:56.796 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.0 MB/s | 2.2 MB 00:01 2024-04-22T20:58:56.855 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-22T20:58:56.870 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-22T20:58:56.870 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-22T20:58:56.952 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-22T20:58:56.953 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-22T20:58:57.535 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-22T20:58:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T20:58:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T20:58:57.826 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T20:58:57.863 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T20:58:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T20:58:58.006 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T20:58:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T20:58:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T20:58:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T20:58:58.929 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T20:58:58.954 INFO:teuthology.orchestra.run.smithi038.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.7 MB/s | 16 MB 00:05 2024-04-22T20:58:58.955 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:58:58.956 INFO:teuthology.orchestra.run.smithi038.stdout:Total 4.8 MB/s | 38 MB 00:07 2024-04-22T20:58:58.962 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T20:58:59.007 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T20:58:59.032 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:17.2.7-884.g70d0cb4b.el7.noarch 13/13 2024-04-22T20:58:59.322 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T20:58:59.349 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T20:58:59.350 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T20:58:59.754 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-884.g70d0cb4b.el7.noarch 13/13 2024-04-22T20:58:59.754 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:17.2.7-884.g70d0cb4b.el7.noarch 1/13 2024-04-22T20:58:59.754 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T20:58:59.754 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T20:58:59.754 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T20:58:59.754 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T20:58:59.754 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T20:58:59.755 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T20:58:59.755 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T20:58:59.755 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T20:58:59.755 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T20:58:59.755 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T20:58:59.755 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T20:58:59.782 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T20:58:59.782 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T20:59:00.053 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T20:59:00.054 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:00.054 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-22T20:59:00.054 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:17.2.7-884.g70d0cb4b.el7.noarch 2024-04-22T20:59:00.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T20:59:00.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T20:59:00.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T20:59:00.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T20:59:00.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T20:59:00.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T20:59:00.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T20:59:00.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T20:59:00.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T20:59:00.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T20:59:00.055 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T20:59:00.055 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T20:59:00.055 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:00.055 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:59:00.300 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T20:59:00.566 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T20:59:00.617 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T20:59:00.661 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T20:59:00.764 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T20:59:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T20:59:00.807 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:18 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:59:00.926 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:59:00.927 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:00.927 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-22T20:59:00.927 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:00.927 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-22T20:59:00.927 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:17.2.7-884.g70d0cb4b.el7 ceph-noarch 123 k 2024-04-22T20:59:00.927 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-22T20:59:00.928 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T20:59:00.928 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T20:59:00.928 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T20:59:00.928 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T20:59:00.928 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T20:59:00.928 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:00.928 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:59:00.928 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:00.928 INFO:teuthology.orchestra.run.smithi098.stdout:Install 6 Packages 2024-04-22T20:59:00.928 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:00.929 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 7.0 M 2024-04-22T20:59:00.929 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 33 M 2024-04-22T20:59:00.929 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-22T20:59:00.988 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T20:59:01.011 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T20:59:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T20:59:01.272 INFO:teuthology.orchestra.run.smithi098.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.0 MB/s | 248 kB 00:00 2024-04-22T20:59:01.406 INFO:teuthology.orchestra.run.smithi098.stdout:(2/6): ceph-mgr-cephadm-17.2.7-884.g70d0cb4b.el 484 kB/s | 123 kB 00:00 2024-04-22T20:59:01.514 INFO:teuthology.orchestra.run.smithi098.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-04-22T20:59:01.714 INFO:teuthology.orchestra.run.smithi098.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 188 kB/s | 58 kB 00:00 2024-04-22T20:59:01.756 INFO:teuthology.orchestra.run.smithi098.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 211 kB/s | 51 kB 00:00 2024-04-22T20:59:01.776 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T20:59:01.815 INFO:teuthology.orchestra.run.smithi098.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-04-22T20:59:01.816 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:59:01.816 INFO:teuthology.orchestra.run.smithi098.stdout:Total 7.8 MB/s | 7.0 MB 00:00 2024-04-22T20:59:01.887 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-22T20:59:01.898 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-22T20:59:01.898 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-22T20:59:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T20:59:01.993 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-22T20:59:01.993 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-22T20:59:02.298 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-22T20:59:02.465 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T20:59:02.517 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T20:59:03.042 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T20:59:03.129 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T20:59:03.184 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T20:59:03.212 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 6/6 2024-04-22T20:59:03.225 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T20:59:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 6/6 2024-04-22T20:59:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 1/6 2024-04-22T20:59:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T20:59:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T20:59:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T20:59:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T20:59:04.309 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T20:59:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T20:59:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:04.318 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-22T20:59:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 2024-04-22T20:59:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T20:59:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T20:59:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T20:59:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T20:59:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T20:59:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:04.319 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:59:04.337 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-884.g70d0cb 12/12 2024-04-22T20:59:04.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-fuse 2024-04-22T20:59:05.063 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:23 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:59:05.183 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:59:05.183 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:05.183 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-22T20:59:05.183 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:05.183 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-22T20:59:05.183 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 812 k 2024-04-22T20:59:05.183 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-22T20:59:05.184 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T20:59:05.184 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:05.184 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:59:05.184 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:05.184 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-04-22T20:59:05.185 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:05.185 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 892 k 2024-04-22T20:59:05.185 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 2.6 M 2024-04-22T20:59:05.185 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-22T20:59:05.718 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 312 kB/s | 80 kB 00:00 2024-04-22T20:59:05.751 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): ceph-fuse-17.2.7-884.g70d0cb4b.el7.x86_6 2.7 MB/s | 812 kB 00:00 2024-04-22T20:59:05.752 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:59:05.752 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.5 MB/s | 892 kB 00:00 2024-04-22T20:59:05.764 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-22T20:59:05.786 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-22T20:59:05.786 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-22T20:59:05.863 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-22T20:59:05.863 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-22T20:59:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-22T20:59:06.189 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T20:59:06.249 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:17.2.7-884.g70d0cb4b.el7.x86_64 2/2 2024-04-22T20:59:06.877 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-884.g70d0cb 12/12 2024-04-22T20:59:06.877 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-884.g70d0cb 1/12 2024-04-22T20:59:06.877 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T20:59:06.877 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T20:59:06.877 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T20:59:06.877 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T20:59:06.877 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T20:59:06.877 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T20:59:06.878 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T20:59:06.878 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T20:59:06.878 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T20:59:06.878 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T20:59:06.878 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-fuse-2:17.2.7-884.g70d0cb4b.el7.x86_64 2/2 2024-04-22T20:59:06.878 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/2 2024-04-22T20:59:07.136 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T20:59:07.136 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:07.136 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-22T20:59:07.136 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:17.2.7-884.g70d0cb4b.el7.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T20:59:07.136 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:07.137 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:59:07.265 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T20:59:07.265 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:07.266 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-22T20:59:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:17.2.7-884.g70d0cb4b.el7.noarch 2024-04-22T20:59:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T20:59:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T20:59:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T20:59:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T20:59:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T20:59:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T20:59:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T20:59:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T20:59:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T20:59:07.267 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T20:59:07.267 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T20:59:07.267 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:07.267 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:59:07.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados-devel 2024-04-22T20:59:07.642 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-rook 2024-04-22T20:59:07.952 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:25 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:59:08.073 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:59:08.073 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:08.074 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-04-22T20:59:08.074 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:08.074 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-22T20:59:08.074 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 126 k 2024-04-22T20:59:08.074 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:08.074 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:59:08.074 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:08.074 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-04-22T20:59:08.074 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:08.075 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 126 k 2024-04-22T20:59:08.075 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 448 k 2024-04-22T20:59:08.075 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-22T20:59:08.179 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:25 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:59:08.273 INFO:teuthology.orchestra.run.smithi098.stdout:librados-devel-17.2.7-884.g70d0cb4b.el7.x86_64. 636 kB/s | 126 kB 00:00 2024-04-22T20:59:08.274 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:59:08.274 INFO:teuthology.orchestra.run.smithi098.stdout:Total 631 kB/s | 126 kB 00:00 2024-04-22T20:59:08.274 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-22T20:59:08.280 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-22T20:59:08.280 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-22T20:59:08.306 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:59:08.307 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:08.307 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-04-22T20:59:08.307 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:08.307 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-22T20:59:08.307 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:17.2.7-884.g70d0cb4b.el7 ceph-noarch 47 k 2024-04-22T20:59:08.307 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-04-22T20:59:08.307 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T20:59:08.307 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T20:59:08.307 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T20:59:08.307 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T20:59:08.308 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T20:59:08.308 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T20:59:08.308 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T20:59:08.308 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T20:59:08.308 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T20:59:08.308 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T20:59:08.308 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T20:59:08.308 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T20:59:08.308 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:08.308 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:59:08.308 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:08.308 INFO:teuthology.orchestra.run.smithi038.stdout:Install 13 Packages 2024-04-22T20:59:08.308 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:08.309 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 2.2 M 2024-04-22T20:59:08.309 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 26 M 2024-04-22T20:59:08.309 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T20:59:08.342 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-22T20:59:08.342 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-22T20:59:08.492 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-22T20:59:08.577 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:08.781 INFO:teuthology.orchestra.run.smithi038.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 249 kB/s | 19 kB 00:00 2024-04-22T20:59:08.831 INFO:teuthology.orchestra.run.smithi038.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 202 kB/s | 26 kB 00:00 2024-04-22T20:59:08.914 INFO:teuthology.orchestra.run.smithi038.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-04-22T20:59:08.940 INFO:teuthology.orchestra.run.smithi038.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-04-22T20:59:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados-devel-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:08.973 INFO:teuthology.orchestra.run.smithi038.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.6 MB/s | 279 kB 00:00 2024-04-22T20:59:08.990 INFO:teuthology.orchestra.run.smithi038.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-04-22T20:59:09.066 INFO:teuthology.orchestra.run.smithi038.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 199 kB/s | 15 kB 00:00 2024-04-22T20:59:09.082 INFO:teuthology.orchestra.run.smithi038.stdout:(8/13): ceph-mgr-rook-17.2.7-884.g70d0cb4b.el7. 125 kB/s | 47 kB 00:00 2024-04-22T20:59:09.099 INFO:teuthology.orchestra.run.smithi038.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 255 kB/s | 32 kB 00:00 2024-04-22T20:59:09.183 INFO:teuthology.orchestra.run.smithi038.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 709 kB/s | 59 kB 00:00 2024-04-22T20:59:09.208 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:09.209 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:09.209 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-22T20:59:09.209 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:59:09.209 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:09.209 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:59:09.250 INFO:teuthology.orchestra.run.smithi038.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 1.1 MB/s | 203 kB 00:00 2024-04-22T20:59:09.275 INFO:teuthology.orchestra.run.smithi038.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 976 kB/s | 90 kB 00:00 2024-04-22T20:59:09.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs2 2024-04-22T20:59:09.759 INFO:teuthology.orchestra.run.smithi038.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2024-04-22T20:59:09.759 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:59:09.760 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.5 MB/s | 2.2 MB 00:01 2024-04-22T20:59:09.831 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T20:59:09.847 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T20:59:09.847 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T20:59:09.940 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T20:59:09.941 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T20:59:09.973 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:27 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:59:10.055 INFO:teuthology.orchestra.run.smithi098.stdout:Package libcephfs2-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:59:10.096 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:59:10.097 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-22T20:59:10.097 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:59:10.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs-devel 2024-04-22T20:59:10.517 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T20:59:10.676 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:28 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:59:10.689 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T20:59:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T20:59:10.795 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:59:10.796 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:10.796 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-04-22T20:59:10.796 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:10.796 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-22T20:59:10.796 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 25 k 2024-04-22T20:59:10.796 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:10.797 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:59:10.797 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:10.797 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-04-22T20:59:10.797 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:10.797 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 25 k 2024-04-22T20:59:10.797 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 110 k 2024-04-22T20:59:10.797 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-22T20:59:10.827 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T20:59:10.863 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T20:59:10.905 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T20:59:10.978 INFO:teuthology.orchestra.run.smithi098.stdout:libcephfs-devel-17.2.7-884.g70d0cb4b.el7.x86_64 137 kB/s | 25 kB 00:00 2024-04-22T20:59:10.979 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:59:10.979 INFO:teuthology.orchestra.run.smithi098.stdout:Total 136 kB/s | 25 kB 00:00 2024-04-22T20:59:10.979 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-22T20:59:10.982 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-22T20:59:10.982 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-22T20:59:10.984 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-22T20:59:10.984 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-22T20:59:11.001 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T20:59:11.072 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-22T20:59:11.113 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T20:59:11.165 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T20:59:11.172 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:11.497 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:11.691 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:11.691 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:11.691 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-22T20:59:11.691 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:59:11.691 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:11.692 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:59:11.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados2 2024-04-22T20:59:11.929 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T20:59:11.979 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T20:59:12.013 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T20:59:12.060 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T20:59:12.082 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:17.2.7-884.g70d0cb4b.el7.noarch 13/13 2024-04-22T20:59:12.366 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:30 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:59:12.444 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:59:12.484 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:59:12.485 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-22T20:59:12.485 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:59:12.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librbd1 2024-04-22T20:59:12.885 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-884.g70d0cb4b.el7.noarch 13/13 2024-04-22T20:59:12.885 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:17.2.7-884.g70d0cb4b.el7.noarch 1/13 2024-04-22T20:59:12.885 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T20:59:12.885 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T20:59:12.885 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T20:59:12.885 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T20:59:12.886 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T20:59:12.886 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T20:59:12.886 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T20:59:12.886 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T20:59:12.886 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T20:59:12.886 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T20:59:12.886 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T20:59:13.061 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:31 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:59:13.141 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:59:13.181 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:59:13.182 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-22T20:59:13.182 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:59:13.213 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T20:59:13.213 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:13.213 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-22T20:59:13.213 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:17.2.7-884.g70d0cb4b.el7.noarch 2024-04-22T20:59:13.213 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T20:59:13.213 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T20:59:13.213 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T20:59:13.213 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T20:59:13.213 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T20:59:13.213 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T20:59:13.213 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T20:59:13.214 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T20:59:13.214 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T20:59:13.214 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T20:59:13.214 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T20:59:13.214 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T20:59:13.214 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:13.214 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:59:13.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rados 2024-04-22T20:59:13.441 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T20:59:13.745 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:31 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:59:13.824 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rados-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:59:13.865 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:59:13.866 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-22T20:59:13.867 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:59:13.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rgw 2024-04-22T20:59:13.973 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:30 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:59:14.098 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:59:14.098 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-04-22T20:59:14.099 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:14.099 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-22T20:59:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:17.2.7-884.g70d0cb4b.el7 ceph-noarch 123 k 2024-04-22T20:59:14.099 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-04-22T20:59:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T20:59:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T20:59:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T20:59:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T20:59:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T20:59:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:14.099 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:59:14.099 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:14.100 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2024-04-22T20:59:14.100 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:14.100 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 7.0 M 2024-04-22T20:59:14.100 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 33 M 2024-04-22T20:59:14.100 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T20:59:14.435 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:32 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:59:14.513 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rgw-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:59:14.553 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:59:14.554 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-22T20:59:14.554 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:59:14.615 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-cephfs 2024-04-22T20:59:14.699 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): ceph-mgr-cephadm-17.2.7-884.g70d0cb4b.el 312 kB/s | 123 kB 00:00 2024-04-22T20:59:14.900 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 416 kB/s | 248 kB 00:00 2024-04-22T20:59:15.025 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-04-22T20:59:15.059 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 366 kB/s | 58 kB 00:00 2024-04-22T20:59:15.092 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 763 kB/s | 51 kB 00:00 2024-04-22T20:59:15.128 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:33 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:59:15.207 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-cephfs-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:59:15.247 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:59:15.248 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-22T20:59:15.248 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:59:15.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rbd 2024-04-22T20:59:15.808 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:33 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:59:15.888 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rbd-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:59:15.929 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:59:15.930 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-22T20:59:15.930 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:59:15.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-fuse 2024-04-22T20:59:16.495 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:34 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:59:16.613 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:59:16.614 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:16.614 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-04-22T20:59:16.614 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:16.614 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-22T20:59:16.614 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 82 k 2024-04-22T20:59:16.615 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:16.615 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:59:16.615 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:16.615 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-04-22T20:59:16.615 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:16.615 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 82 k 2024-04-22T20:59:16.615 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 222 k 2024-04-22T20:59:16.615 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-22T20:59:16.742 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.4 MB/s | 6.0 MB 00:02 2024-04-22T20:59:16.745 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:59:16.745 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.6 MB/s | 7.0 MB 00:02 2024-04-22T20:59:16.818 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T20:59:16.830 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T20:59:16.830 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T20:59:16.831 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-fuse-17.2.7-884.g70d0cb4b.el7.x86_64.rpm 379 kB/s | 82 kB 00:00 2024-04-22T20:59:16.831 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:59:16.832 INFO:teuthology.orchestra.run.smithi098.stdout:Total 376 kB/s | 82 kB 00:00 2024-04-22T20:59:16.832 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-22T20:59:16.838 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-22T20:59:16.838 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-22T20:59:16.900 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-22T20:59:16.901 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-22T20:59:16.931 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T20:59:16.933 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T20:59:17.042 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-22T20:59:17.142 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:17.243 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T20:59:17.423 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T20:59:17.479 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T20:59:17.485 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-fuse-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:17.717 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-22T20:59:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:59:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:17.717 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:59:17.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-mirror 2024-04-22T20:59:18.035 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T20:59:18.117 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T20:59:18.179 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T20:59:18.215 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 6/6 2024-04-22T20:59:18.472 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:36 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:59:18.592 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:59:18.593 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:18.593 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-04-22T20:59:18.593 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:18.593 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-22T20:59:18.593 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 2.9 M 2024-04-22T20:59:18.593 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:18.594 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:59:18.594 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:18.594 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-04-22T20:59:18.594 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:18.594 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 2.9 M 2024-04-22T20:59:18.594 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 12 M 2024-04-22T20:59:18.594 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-22T20:59:18.952 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 6/6 2024-04-22T20:59:18.953 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 1/6 2024-04-22T20:59:18.953 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T20:59:18.953 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T20:59:18.953 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T20:59:18.953 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T20:59:19.200 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T20:59:19.200 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:19.200 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-22T20:59:19.200 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:17.2.7-884.g70d0cb4b.el7.noarch 2024-04-22T20:59:19.200 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T20:59:19.200 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T20:59:19.200 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T20:59:19.200 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T20:59:19.200 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T20:59:19.200 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:19.200 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:59:19.417 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-fuse 2024-04-22T20:59:19.610 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-mirror-17.2.7-884.g70d0cb4b.el7.x86_64.rpm 2.9 MB/s | 2.9 MB 00:01 2024-04-22T20:59:19.611 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:59:19.611 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.9 MB/s | 2.9 MB 00:01 2024-04-22T20:59:19.611 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-22T20:59:19.621 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-22T20:59:19.621 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-22T20:59:19.699 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-22T20:59:19.699 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-22T20:59:19.950 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:36 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:59:20.073 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:59:20.074 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:20.074 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-04-22T20:59:20.074 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:20.074 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-22T20:59:20.074 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 812 k 2024-04-22T20:59:20.074 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-04-22T20:59:20.074 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T20:59:20.074 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:20.074 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:59:20.074 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:20.074 INFO:teuthology.orchestra.run.smithi038.stdout:Install 2 Packages 2024-04-22T20:59:20.075 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:20.075 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 892 k 2024-04-22T20:59:20.075 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 2.6 M 2024-04-22T20:59:20.075 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T20:59:20.113 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-22T20:59:20.186 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:20.209 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T20:59:20.209 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T20:59:20.210 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T20:59:20.210 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T20:59:20.210 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:20.678 INFO:teuthology.orchestra.run.smithi038.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 239 kB/s | 80 kB 00:00 2024-04-22T20:59:20.919 INFO:teuthology.orchestra.run.smithi038.stdout:(2/2): ceph-fuse-17.2.7-884.g70d0cb4b.el7.x86_6 1.4 MB/s | 812 kB 00:00 2024-04-22T20:59:20.920 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:59:20.920 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.0 MB/s | 892 kB 00:00 2024-04-22T20:59:20.933 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T20:59:20.958 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T20:59:20.958 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T20:59:21.041 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T20:59:21.041 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T20:59:21.226 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T20:59:21.249 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:21.249 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:21.249 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-22T20:59:21.249 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:59:21.249 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:21.249 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:59:21.361 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T20:59:21.436 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:17.2.7-884.g70d0cb4b.el7.x86_64 2/2 2024-04-22T20:59:21.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-nbd 2024-04-22T20:59:22.031 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:40 ago on Mon 22 Apr 2024 08:57:42 PM UTC. 2024-04-22T20:59:22.034 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-fuse-2:17.2.7-884.g70d0cb4b.el7.x86_64 2/2 2024-04-22T20:59:22.034 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/2 2024-04-22T20:59:22.154 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-22T20:59:22.155 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-04-22T20:59:22.156 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:22.156 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-22T20:59:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 165 k 2024-04-22T20:59:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:22.156 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-22T20:59:22.156 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-22T20:59:22.157 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-04-22T20:59:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:22.157 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 165 k 2024-04-22T20:59:22.157 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 477 k 2024-04-22T20:59:22.157 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-22T20:59:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T20:59:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:22.257 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-22T20:59:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:17.2.7-884.g70d0cb4b.el7.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T20:59:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:22.258 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:59:22.473 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-nbd-17.2.7-884.g70d0cb4b.el7.x86_64.rpm 521 kB/s | 165 kB 00:00 2024-04-22T20:59:22.474 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:59:22.474 INFO:teuthology.orchestra.run.smithi098.stdout:Total 518 kB/s | 165 kB 00:00 2024-04-22T20:59:22.474 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-22T20:59:22.481 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-22T20:59:22.482 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-22T20:59:22.537 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados-devel 2024-04-22T20:59:22.546 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-22T20:59:22.546 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-22T20:59:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-22T20:59:22.795 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:23.076 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:40 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:59:23.202 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:59:23.202 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:23.203 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-04-22T20:59:23.203 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:23.203 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-22T20:59:23.203 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 126 k 2024-04-22T20:59:23.203 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:23.203 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:59:23.203 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:23.203 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-04-22T20:59:23.203 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:23.203 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 126 k 2024-04-22T20:59:23.203 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 448 k 2024-04-22T20:59:23.204 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T20:59:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-nbd-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:23.490 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:23.491 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-22T20:59:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:59:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T20:59:23.491 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-22T20:59:23.516 INFO:teuthology.orchestra.run.smithi038.stdout:librados-devel-17.2.7-884.g70d0cb4b.el7.x86_64. 405 kB/s | 126 kB 00:00 2024-04-22T20:59:23.517 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:59:23.517 INFO:teuthology.orchestra.run.smithi038.stdout:Total 403 kB/s | 126 kB 00:00 2024-04-22T20:59:23.517 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T20:59:23.523 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T20:59:23.523 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T20:59:23.589 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T20:59:23.589 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T20:59:23.724 DEBUG:teuthology.parallel:result is None 2024-04-22T20:59:23.763 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T20:59:23.898 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:24.319 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados-devel-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:24.550 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:24.551 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:24.551 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-22T20:59:24.551 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:59:24.551 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:24.551 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:59:24.793 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs2 2024-04-22T20:59:25.332 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:42 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:59:25.413 INFO:teuthology.orchestra.run.smithi038.stdout:Package libcephfs2-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:59:25.456 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:59:25.457 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-04-22T20:59:25.457 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:59:25.528 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs-devel 2024-04-22T20:59:26.058 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:43 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:59:26.182 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:59:26.183 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:26.183 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-04-22T20:59:26.183 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:26.183 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-22T20:59:26.183 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 25 k 2024-04-22T20:59:26.183 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:26.183 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:59:26.183 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:26.183 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-04-22T20:59:26.183 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:26.183 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 25 k 2024-04-22T20:59:26.184 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 110 k 2024-04-22T20:59:26.184 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T20:59:26.353 INFO:teuthology.orchestra.run.smithi038.stdout:libcephfs-devel-17.2.7-884.g70d0cb4b.el7.x86_64 147 kB/s | 25 kB 00:00 2024-04-22T20:59:26.354 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:59:26.354 INFO:teuthology.orchestra.run.smithi038.stdout:Total 146 kB/s | 25 kB 00:00 2024-04-22T20:59:26.354 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T20:59:26.358 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T20:59:26.358 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T20:59:26.359 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T20:59:26.360 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T20:59:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T20:59:26.552 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:27.088 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:27.088 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:27.089 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-22T20:59:27.089 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:59:27.089 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:27.089 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:59:27.261 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados2 2024-04-22T20:59:27.792 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:44 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:59:27.873 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:59:27.916 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:59:27.917 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-04-22T20:59:27.917 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:59:27.987 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librbd1 2024-04-22T20:59:28.529 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:45 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:59:28.612 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:59:28.655 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:59:28.656 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-04-22T20:59:28.656 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:59:28.718 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rados 2024-04-22T20:59:29.249 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:46 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:59:29.331 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rados-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:59:29.373 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:59:29.374 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-04-22T20:59:29.375 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:59:29.454 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rgw 2024-04-22T20:59:29.987 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:46 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:59:30.068 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rgw-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:59:30.114 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:59:30.115 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-04-22T20:59:30.115 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:59:30.184 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-cephfs 2024-04-22T20:59:30.724 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:47 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:59:30.807 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-cephfs-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:59:30.850 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:59:30.851 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-04-22T20:59:30.851 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:59:30.926 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rbd 2024-04-22T20:59:31.458 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:48 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:59:31.540 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rbd-2:17.2.7-884.g70d0cb4b.el7.x86_64 is already installed. 2024-04-22T20:59:31.583 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:59:31.584 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-04-22T20:59:31.584 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:59:31.654 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-fuse 2024-04-22T20:59:32.188 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:49 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:59:32.311 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:59:32.312 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:32.312 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-04-22T20:59:32.312 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:32.312 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-22T20:59:32.313 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 82 k 2024-04-22T20:59:32.313 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:32.313 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:59:32.313 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:32.313 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-04-22T20:59:32.313 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:32.313 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 82 k 2024-04-22T20:59:32.313 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 222 k 2024-04-22T20:59:32.313 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T20:59:32.598 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-fuse-17.2.7-884.g70d0cb4b.el7.x86_64.rpm 287 kB/s | 82 kB 00:00 2024-04-22T20:59:32.599 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:59:32.599 INFO:teuthology.orchestra.run.smithi038.stdout:Total 286 kB/s | 82 kB 00:00 2024-04-22T20:59:32.599 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T20:59:32.606 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T20:59:32.607 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T20:59:32.674 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T20:59:32.674 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T20:59:32.821 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T20:59:32.921 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:33.275 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-fuse-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:33.531 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:33.531 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:33.531 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-22T20:59:33.531 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:59:33.532 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:33.532 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:59:33.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-mirror 2024-04-22T20:59:34.312 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:51 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:59:34.441 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:59:34.442 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-04-22T20:59:34.442 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:34.442 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-22T20:59:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 2.9 M 2024-04-22T20:59:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:34.442 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:59:34.442 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:34.442 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-04-22T20:59:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:34.443 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 2.9 M 2024-04-22T20:59:34.443 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 12 M 2024-04-22T20:59:34.443 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T20:59:34.902 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-mirror-17.2.7-884.g70d0cb4b.el7.x86_64.rpm 6.4 MB/s | 2.9 MB 00:00 2024-04-22T20:59:34.903 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:59:34.903 INFO:teuthology.orchestra.run.smithi038.stdout:Total 6.4 MB/s | 2.9 MB 00:00 2024-04-22T20:59:34.904 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T20:59:34.921 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T20:59:34.921 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T20:59:35.013 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T20:59:35.014 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T20:59:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T20:59:35.612 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:35.638 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:35.638 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T20:59:35.638 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T20:59:35.638 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T20:59:35.638 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T20:59:35.638 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:36.630 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:36.630 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:36.630 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-22T20:59:36.630 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:59:36.630 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:36.630 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:59:36.891 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-nbd 2024-04-22T20:59:37.429 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:54 ago on Mon 22 Apr 2024 08:57:43 PM UTC. 2024-04-22T20:59:37.554 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T20:59:37.554 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:37.554 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-04-22T20:59:37.554 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:37.554 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-22T20:59:37.555 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:17.2.7-884.g70d0cb4b.el7 ceph 165 k 2024-04-22T20:59:37.555 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:37.555 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T20:59:37.555 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T20:59:37.555 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-04-22T20:59:37.555 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:37.555 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 165 k 2024-04-22T20:59:37.555 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 477 k 2024-04-22T20:59:37.555 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T20:59:37.782 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-nbd-17.2.7-884.g70d0cb4b.el7.x86_64.rpm 727 kB/s | 165 kB 00:00 2024-04-22T20:59:37.783 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T20:59:37.783 INFO:teuthology.orchestra.run.smithi038.stdout:Total 722 kB/s | 165 kB 00:00 2024-04-22T20:59:37.783 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T20:59:37.791 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T20:59:37.791 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T20:59:37.859 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T20:59:37.859 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T20:59:38.012 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T20:59:38.113 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:38.459 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-nbd-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:38.688 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:17.2.7-884.g70d0cb4b.el7.x86_64 1/1 2024-04-22T20:59:38.688 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:38.688 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-22T20:59:38.688 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:17.2.7-884.g70d0cb4b.el7.x86_64 2024-04-22T20:59:38.688 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T20:59:38.689 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T20:59:38.928 DEBUG:teuthology.parallel:result is None 2024-04-22T20:59:38.928 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T20:59:38.928 INFO:teuthology.packaging:ref: None 2024-04-22T20:59:38.928 INFO:teuthology.packaging:tag: None 2024-04-22T20:59:38.928 INFO:teuthology.packaging:branch: quincy 2024-04-22T20:59:38.928 INFO:teuthology.packaging:sha1: 56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-22T20:59:38.929 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-04-22T20:59:39.169 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T20:59:39.191 INFO:teuthology.orchestra.run.smithi038.stdout:17.2.7-884.g70d0cb4b.el7 2024-04-22T20:59:39.192 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-884.g70d0cb4b.el7 2024-04-22T20:59:39.192 INFO:teuthology.task.install:The correct ceph version 17.2.7-884.g70d0cb4b is installed. 2024-04-22T20:59:39.194 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T20:59:39.194 INFO:teuthology.packaging:ref: None 2024-04-22T20:59:39.194 INFO:teuthology.packaging:tag: None 2024-04-22T20:59:39.194 INFO:teuthology.packaging:branch: quincy 2024-04-22T20:59:39.194 INFO:teuthology.packaging:sha1: 56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-22T20:59:39.194 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-04-22T20:59:39.424 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T20:59:39.449 INFO:teuthology.orchestra.run.smithi098.stdout:17.2.7-884.g70d0cb4b.el7 2024-04-22T20:59:39.449 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-884.g70d0cb4b.el7 2024-04-22T20:59:39.450 INFO:teuthology.task.install:The correct ceph version 17.2.7-884.g70d0cb4b is installed. 2024-04-22T20:59:39.451 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T20:59:39.452 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T20:59:39.452 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T20:59:39.479 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T20:59:39.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T20:59:39.520 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T20:59:39.520 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T20:59:39.520 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T20:59:39.546 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T20:59:39.612 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T20:59:39.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T20:59:39.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T20:59:39.709 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T20:59:39.709 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T20:59:39.709 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T20:59:39.735 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T20:59:39.800 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T20:59:39.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T20:59:39.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T20:59:39.895 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T20:59:39.895 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T20:59:39.896 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T20:59:39.920 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T20:59:39.986 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T20:59:39.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T20:59:40.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T20:59:40.085 INFO:teuthology.run_tasks:Running task print... 2024-04-22T20:59:40.096 INFO:teuthology.task.print:**** done install task... 2024-04-22T20:59:40.096 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-22T20:59:40.196 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '56e81df3aeb98b717efe2ab0537a8c60249f95f7'} 2024-04-22T20:59:40.196 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-04-22T20:59:40.197 INFO:tasks.cephadm:Cluster fsid is 3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T20:59:40.197 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-22T20:59:40.197 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-22T20:59:40.197 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi038': '172.21.15.38', 'mon.smithi098': '172.21.15.98'} 2024-04-22T20:59:40.197 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-22T20:59:40.197 DEBUG:teuthology.orchestra.run.smithi038:> sudo hostname $(hostname -s) 2024-04-22T20:59:40.225 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2024-04-22T20:59:40.253 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-04-22T20:59:40.253 DEBUG:teuthology.orchestra.run.smithi038:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T20:59:40.480 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Apr 22 20:59 /home/ubuntu/cephtest/cephadm 2024-04-22T20:59:40.481 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T20:59:40.698 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Apr 22 20:59 /home/ubuntu/cephtest/cephadm 2024-04-22T20:59:40.699 DEBUG:teuthology.orchestra.run.smithi038:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T20:59:40.718 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T20:59:40.746 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-04-22T20:59:40.746 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-04-22T20:59:40.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-04-22T20:59:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-04-22T20:59:42.422 INFO:teuthology.orchestra.run.smithi038.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-04-22T21:00:09.734 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-22T21:00:09.734 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 17.2.7-884-g70d0cb4b (70d0cb4b9d576828afb0a05dcf8732894d343ece) quincy (stable)", 2024-04-22T21:00:09.734 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "cafd8ddac8fa176bf00aeacab8beaeeffcb0c5261d2d415c33683cb8bed0534e", 2024-04-22T21:00:09.734 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2024-04-22T21:00:09.734 INFO:teuthology.orchestra.run.smithi098.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:3890cd726bd5e73ae623dabdf4ed76dd5df81c936b8bd8d761026babda969466" 2024-04-22T21:00:09.734 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-22T21:00:09.734 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-22T21:00:11.003 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T21:00:11.003 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_version": "ceph version 17.2.7-884-g70d0cb4b (70d0cb4b9d576828afb0a05dcf8732894d343ece) quincy (stable)", 2024-04-22T21:00:11.003 INFO:teuthology.orchestra.run.smithi038.stdout: "image_id": "cafd8ddac8fa176bf00aeacab8beaeeffcb0c5261d2d415c33683cb8bed0534e", 2024-04-22T21:00:11.003 INFO:teuthology.orchestra.run.smithi038.stdout: "repo_digests": [ 2024-04-22T21:00:11.003 INFO:teuthology.orchestra.run.smithi038.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:3890cd726bd5e73ae623dabdf4ed76dd5df81c936b8bd8d761026babda969466" 2024-04-22T21:00:11.003 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T21:00:11.003 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T21:00:11.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph 2024-04-22T21:00:11.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2024-04-22T21:00:11.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /etc/ceph 2024-04-22T21:00:11.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2024-04-22T21:00:11.153 INFO:tasks.cephadm:Writing seed config... 2024-04-22T21:00:11.155 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-22T21:00:11.155 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-22T21:00:11.155 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-22T21:00:11.155 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-22T21:00:11.155 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-22T21:00:11.155 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-22T21:00:11.155 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-22T21:00:11.155 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-22T21:00:11.155 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-22T21:00:11.155 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-22T21:00:11.155 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-22T21:00:11.155 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-22T21:00:11.156 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-22T21:00:11.156 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-22T21:00:11.156 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-22T21:00:11.156 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-22T21:00:11.156 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-22T21:00:11.156 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-22T21:00:11.156 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-22T21:00:11.156 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-22T21:00:11.156 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-22T21:00:11.156 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-22T21:00:11.156 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-22T21:00:11.156 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-22T21:00:11.156 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-22T21:00:11.156 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-22T21:00:11.156 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-22T21:00:11.157 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-22T21:00:11.157 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-22T21:00:11.157 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-22T21:00:11.157 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-22T21:00:11.157 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-22T21:00:11.157 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-22T21:00:11.157 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-22T21:00:11.157 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-22T21:00:11.157 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-22T21:00:11.157 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-22T21:00:11.157 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-22T21:00:11.157 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-22T21:00:11.157 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-22T21:00:11.157 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-22T21:00:11.157 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-22T21:00:11.158 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T21:00:11.158 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-22T21:00:11.174 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3c5f5352-00eb-11ef-bc93-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-22T21:00:11.174 DEBUG:teuthology.orchestra.run.smithi038:mon.smithi038> sudo journalctl -f -n 0 -u ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@mon.smithi038.service 2024-04-22T21:00:11.216 INFO:tasks.cephadm:Bootstrapping... 2024-04-22T21:00:11.217 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.38 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:00:11.426 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T21:00:11.426 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '3c5f5352-00eb-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.38', '--skip-admin-label'] 2024-04-22T21:00:11.453 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T21:00:11.453 INFO:teuthology.orchestra.run.smithi038.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-22T21:00:11.454 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying podman|docker is present... 2024-04-22T21:00:11.477 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T21:00:11.477 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying lvm2 is present... 2024-04-22T21:00:11.478 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying time synchronization is in place... 2024-04-22T21:00:11.485 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T21:00:11.485 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T21:00:11.492 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T21:00:11.492 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-22T21:00:11.499 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout enabled 2024-04-22T21:00:11.505 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout active 2024-04-22T21:00:11.505 INFO:teuthology.orchestra.run.smithi038.stdout:Unit chronyd.service is enabled and running 2024-04-22T21:00:11.505 INFO:teuthology.orchestra.run.smithi038.stdout:Repeating the final host check... 2024-04-22T21:00:11.529 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T21:00:11.530 INFO:teuthology.orchestra.run.smithi038.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-22T21:00:11.530 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl is present 2024-04-22T21:00:11.530 INFO:teuthology.orchestra.run.smithi038.stdout:lvcreate is present 2024-04-22T21:00:11.536 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T21:00:11.536 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T21:00:11.543 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T21:00:11.543 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-22T21:00:11.549 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout enabled 2024-04-22T21:00:11.556 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout active 2024-04-22T21:00:11.556 INFO:teuthology.orchestra.run.smithi038.stdout:Unit chronyd.service is enabled and running 2024-04-22T21:00:11.556 INFO:teuthology.orchestra.run.smithi038.stdout:Host looks OK 2024-04-22T21:00:11.557 INFO:teuthology.orchestra.run.smithi038.stdout:Cluster fsid: 3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T21:00:11.557 INFO:teuthology.orchestra.run.smithi038.stdout:Acquiring lock 139621004608848 on /run/cephadm/3c5f5352-00eb-11ef-bc93-c7b262605968.lock 2024-04-22T21:00:11.557 INFO:teuthology.orchestra.run.smithi038.stdout:Lock 139621004608848 acquired on /run/cephadm/3c5f5352-00eb-11ef-bc93-c7b262605968.lock 2024-04-22T21:00:11.557 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying IP 172.21.15.38 port 3300 ... 2024-04-22T21:00:11.557 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying IP 172.21.15.38 port 6789 ... 2024-04-22T21:00:11.558 INFO:teuthology.orchestra.run.smithi038.stdout:Base mon IP(s) is [172.21.15.38:3300, 172.21.15.38:6789], mon addrv is [v2:172.21.15.38:3300,v1:172.21.15.38:6789] 2024-04-22T21:00:11.561 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.38 metric 100 2024-04-22T21:00:11.561 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.38 metric 100 2024-04-22T21:00:11.564 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-22T21:00:11.564 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-22T21:00:11.564 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-22T21:00:11.567 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-22T21:00:11.567 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-22T21:00:11.567 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T21:00:11.567 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-22T21:00:11.567 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15fc/64 scope link noprefixroute 2024-04-22T21:00:11.567 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T21:00:11.568 INFO:teuthology.orchestra.run.smithi038.stdout:Mon IP `172.21.15.38` is in CIDR network `172.21.0.0/20` 2024-04-22T21:00:11.568 INFO:teuthology.orchestra.run.smithi038.stdout:Mon IP `172.21.15.38` is in CIDR network `172.21.0.0/20` 2024-04-22T21:00:11.568 INFO:teuthology.orchestra.run.smithi038.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-22T21:00:11.568 INFO:teuthology.orchestra.run.smithi038.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-22T21:00:11.569 INFO:teuthology.orchestra.run.smithi038.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-04-22T21:00:14.752 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout cafd8ddac8fa176bf00aeacab8beaeeffcb0c5261d2d415c33683cb8bed0534e 2024-04-22T21:00:14.753 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-04-22T21:00:14.753 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Getting image source signatures 2024-04-22T21:00:14.753 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying blob sha256:9977ec9f18cf69a6eaba4ac817ba680893fdb6756bbeabcd26f0bbb94288ba4a 2024-04-22T21:00:14.753 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-22T21:00:14.753 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying config sha256:cafd8ddac8fa176bf00aeacab8beaeeffcb0c5261d2d415c33683cb8bed0534e 2024-04-22T21:00:14.753 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-22T21:00:15.666 INFO:teuthology.orchestra.run.smithi038.stdout:ceph: stdout ceph version 17.2.7-884-g70d0cb4b (70d0cb4b9d576828afb0a05dcf8732894d343ece) quincy (stable) 2024-04-22T21:00:15.667 INFO:teuthology.orchestra.run.smithi038.stdout:Ceph version: ceph version 17.2.7-884-g70d0cb4b (70d0cb4b9d576828afb0a05dcf8732894d343ece) quincy (stable) 2024-04-22T21:00:15.667 INFO:teuthology.orchestra.run.smithi038.stdout:Extracting ceph user uid/gid from container image... 2024-04-22T21:00:16.676 INFO:teuthology.orchestra.run.smithi038.stdout:stat: stdout 167 167 2024-04-22T21:00:16.676 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial keys... 2024-04-22T21:00:17.574 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQDhzyZma9NOABAAD47fUnm3qx/Ztio1AhbK5w== 2024-04-22T21:00:18.456 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQDhzyZmxppbNBAAZWMqn8WfB+zORABTAM9GOA== 2024-04-22T21:00:19.389 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQDizyZmUhLdLRAAUSoTApL7+DafJbwgjC+wHA== 2024-04-22T21:00:19.390 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial monmap... 2024-04-22T21:00:20.225 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T21:00:20.226 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-04-22T21:00:20.226 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T21:00:20.226 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T21:00:20.226 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool for smithi038 [v2:172.21.15.38:3300,v1:172.21.15.38:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T21:00:20.226 INFO:teuthology.orchestra.run.smithi038.stdout:setting min_mon_release = octopus 2024-04-22T21:00:20.226 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: set fsid to 3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T21:00:20.226 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T21:00:20.226 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:00:20.226 INFO:teuthology.orchestra.run.smithi038.stdout:Creating mon... 2024-04-22T21:00:21.138 INFO:teuthology.orchestra.run.smithi038.stdout:create mon.smithi038 on 2024-04-22T21:00:21.348 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-22T21:00:21.492 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-22T21:00:21.644 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3c5f5352-00eb-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-3c5f5352-00eb-11ef-bc93-c7b262605968.target. 2024-04-22T21:00:21.644 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-3c5f5352-00eb-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-3c5f5352-00eb-11ef-bc93-c7b262605968.target. 2024-04-22T21:00:21.817 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@mon.smithi038 2024-04-22T21:00:21.817 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@mon.smithi038.service: Unit ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@mon.smithi038.service not loaded. 2024-04-22T21:00:21.990 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3c5f5352-00eb-11ef-bc93-c7b262605968.target.wants/ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@mon.smithi038.service → /etc/systemd/system/ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@.service. 2024-04-22T21:00:22.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:22 smithi038 podman[25973]: 2024-04-22 21:00:22.14857124 +0000 UTC m=+0.018015443 image pull cafd8ddac8fa176bf00aeacab8beaeeffcb0c5261d2d415c33683cb8bed0534e quay.ceph.io/ceph-ci/ceph:quincy 2024-04-22T21:00:23.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:23 smithi038 podman[25973]: 2024-04-22 21:00:23.445296254 +0000 UTC m=+1.314740439 container create 11af5584d9d0787d1f015e396c6d00bbf63d7dbc83b18a05076f037bbc302f93 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-mon-smithi038, org.label-schema.schema-version=1.0, RELEASE=quincy-70d0cb4, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-04-22T21:00:25.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:25 smithi038 podman[25973]: 2024-04-22 21:00:25.227428847 +0000 UTC m=+3.096873046 container init 11af5584d9d0787d1f015e396c6d00bbf63d7dbc83b18a05076f037bbc302f93 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-mon-smithi038, RELEASE=quincy-70d0cb4, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-22T21:00:25.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:25 smithi038 podman[25973]: 2024-04-22 21:00:25.231824587 +0000 UTC m=+3.101268773 container start 11af5584d9d0787d1f015e396c6d00bbf63d7dbc83b18a05076f037bbc302f93 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-mon-smithi038, GIT_BRANCH=HEAD, RELEASE=quincy-70d0cb4, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-22T21:00:25.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:25 smithi038 ceph-mon[26004]: mkfs 3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T21:00:25.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:25 smithi038 ceph-mon[26004]: mon.smithi038 is new leader, mons smithi038 in quorum (ranks 0) 2024-04-22T21:00:26.034 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T21:00:26.034 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-04-22T21:00:26.043 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T21:00:26.043 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-22T21:00:26.043 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-04-22T21:00:26.043 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T21:00:26.043 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mon to start... 2024-04-22T21:00:26.044 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mon... 2024-04-22T21:00:26.282 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:26 smithi038 bash[25973]: 11af5584d9d0787d1f015e396c6d00bbf63d7dbc83b18a05076f037bbc302f93 2024-04-22T21:00:26.282 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:26 smithi038 systemd[1]: Started Ceph mon.smithi038 for 3c5f5352-00eb-11ef-bc93-c7b262605968. 2024-04-22T21:00:26.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:26 smithi038 ceph-mon[26004]: mon.smithi038 is new leader, mons smithi038 in quorum (ranks 0) 2024-04-22T21:00:26.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:26 smithi038 ceph-mon[26004]: monmap e1: 1 mons at {smithi038=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-22T21:00:26.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:26 smithi038 ceph-mon[26004]: fsmap 2024-04-22T21:00:26.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:26 smithi038 ceph-mon[26004]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T21:00:26.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:26 smithi038 ceph-mon[26004]: mgrmap e1: no daemons active 2024-04-22T21:00:29.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:29 smithi038 ceph-mon[26004]: from='client.? 172.21.15.38:0/3376070132' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-22T21:00:32.427 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout cluster: 2024-04-22T21:00:32.427 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout id: 3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T21:00:32.427 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-22T21:00:32.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T21:00:32.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout services: 2024-04-22T21:00:32.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi038 (age 3s) 2024-04-22T21:00:32.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-22T21:00:32.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-22T21:00:32.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T21:00:32.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout data: 2024-04-22T21:00:32.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-22T21:00:32.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-22T21:00:32.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-22T21:00:32.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout pgs: 2024-04-22T21:00:32.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T21:00:32.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.007+0000 7fbda95a2700 1 Processor -- start 2024-04-22T21:00:32.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.007+0000 7fbda95a2700 1 -- start start 2024-04-22T21:00:32.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.007+0000 7fbda95a2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbda40712c0 0x7fbda4105360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:32.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.007+0000 7fbda95a2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda40716d0 con 0x7fbda40712c0 2024-04-22T21:00:32.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.008+0000 7fbda3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbda40712c0 0x7fbda4105360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:32.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.008+0000 7fbda3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbda40712c0 0x7fbda4105360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50272/0 (socket says 172.21.15.38:50272) 2024-04-22T21:00:32.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.008+0000 7fbda3fff700 1 -- 172.21.15.38:0/3319181441 learned_addr learned my addr 172.21.15.38:0/3319181441 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:32.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.008+0000 7fbda3fff700 1 -- 172.21.15.38:0/3319181441 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda41058a0 con 0x7fbda40712c0 2024-04-22T21:00:32.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.009+0000 7fbda3fff700 1 --2- 172.21.15.38:0/3319181441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbda40712c0 0x7fbda4105360 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbd940098e0 tx=0x7fbd94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=578b036b70f05ff8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:32.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.009+0000 7fbda2ffd700 1 -- 172.21.15.38:0/3319181441 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd9400b860 con 0x7fbda40712c0 2024-04-22T21:00:32.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.009+0000 7fbda2ffd700 1 -- 172.21.15.38:0/3319181441 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9400b9c0 con 0x7fbda40712c0 2024-04-22T21:00:32.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.009+0000 7fbda2ffd700 1 -- 172.21.15.38:0/3319181441 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd9400bc90 con 0x7fbda40712c0 2024-04-22T21:00:32.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.009+0000 7fbda95a2700 1 -- 172.21.15.38:0/3319181441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbda40712c0 msgr2=0x7fbda4105360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:32.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.009+0000 7fbda95a2700 1 --2- 172.21.15.38:0/3319181441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbda40712c0 0x7fbda4105360 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbd940098e0 tx=0x7fbd94009250 comp rx=0 tx=0).stop 2024-04-22T21:00:32.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.009+0000 7fbda95a2700 1 -- 172.21.15.38:0/3319181441 shutdown_connections 2024-04-22T21:00:32.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.009+0000 7fbda95a2700 1 --2- 172.21.15.38:0/3319181441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbda40712c0 0x7fbda4105360 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:32.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.009+0000 7fbda95a2700 1 -- 172.21.15.38:0/3319181441 >> 172.21.15.38:0/3319181441 conn(0x7fbda40fcdd0 msgr2=0x7fbda40ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:32.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.009+0000 7fbda95a2700 1 -- 172.21.15.38:0/3319181441 shutdown_connections 2024-04-22T21:00:32.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.009+0000 7fbda95a2700 1 -- 172.21.15.38:0/3319181441 wait complete. 2024-04-22T21:00:32.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.010+0000 7fbda95a2700 1 Processor -- start 2024-04-22T21:00:32.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.010+0000 7fbda95a2700 1 -- start start 2024-04-22T21:00:32.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.010+0000 7fbda95a2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbda40712c0 0x7fbda4110d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:32.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.010+0000 7fbda95a2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda4111250 con 0x7fbda40712c0 2024-04-22T21:00:32.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.010+0000 7fbda3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbda40712c0 0x7fbda4110d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:32.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.010+0000 7fbda3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbda40712c0 0x7fbda4110d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50288/0 (socket says 172.21.15.38:50288) 2024-04-22T21:00:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.011+0000 7fbda3fff700 1 -- 172.21.15.38:0/3376070132 learned_addr learned my addr 172.21.15.38:0/3376070132 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.011+0000 7fbda3fff700 1 -- 172.21.15.38:0/3376070132 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd940094f0 con 0x7fbda40712c0 2024-04-22T21:00:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.011+0000 7fbda3fff700 1 --2- 172.21.15.38:0/3376070132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbda40712c0 0x7fbda4110d10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbd94009290 tx=0x7fbd94016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.011+0000 7fbda0ff9700 1 -- 172.21.15.38:0/3376070132 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd9401c5c0 con 0x7fbda40712c0 2024-04-22T21:00:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.011+0000 7fbda0ff9700 1 -- 172.21.15.38:0/3376070132 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9401c740 con 0x7fbda40712c0 2024-04-22T21:00:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.011+0000 7fbda0ff9700 1 -- 172.21.15.38:0/3376070132 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd9401ca50 con 0x7fbda40712c0 2024-04-22T21:00:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.011+0000 7fbda95a2700 1 -- 172.21.15.38:0/3376070132 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda4111450 con 0x7fbda40712c0 2024-04-22T21:00:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.011+0000 7fbda95a2700 1 -- 172.21.15.38:0/3376070132 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda4111fc0 con 0x7fbda40712c0 2024-04-22T21:00:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.011+0000 7fbda0ff9700 1 -- 172.21.15.38:0/3376070132 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fbd9401cbb0 con 0x7fbda40712c0 2024-04-22T21:00:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.011+0000 7fbda0ff9700 1 -- 172.21.15.38:0/3376070132 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fbd9401ac30 con 0x7fbda40712c0 2024-04-22T21:00:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.012+0000 7fbda95a2700 1 -- 172.21.15.38:0/3376070132 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda405f5f0 con 0x7fbda40712c0 2024-04-22T21:00:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.015+0000 7fbda0ff9700 1 -- 172.21.15.38:0/3376070132 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fbd9401ae40 con 0x7fbda40712c0 2024-04-22T21:00:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.087+0000 7fbda95a2700 1 -- 172.21.15.38:0/3376070132 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fbda4112a90 con 0x7fbda40712c0 2024-04-22T21:00:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.087+0000 7fbda0ff9700 1 -- 172.21.15.38:0/3376070132 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+318 (secure 0 0 0) 0x7fbd94025440 con 0x7fbda40712c0 2024-04-22T21:00:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.088+0000 7fbda95a2700 1 -- 172.21.15.38:0/3376070132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbda40712c0 msgr2=0x7fbda4110d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.088+0000 7fbda95a2700 1 --2- 172.21.15.38:0/3376070132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbda40712c0 0x7fbda4110d10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbd94009290 tx=0x7fbd94016000 comp rx=0 tx=0).stop 2024-04-22T21:00:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.088+0000 7fbda95a2700 1 -- 172.21.15.38:0/3376070132 shutdown_connections 2024-04-22T21:00:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.088+0000 7fbda95a2700 1 --2- 172.21.15.38:0/3376070132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbda40712c0 0x7fbda4110d10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.088+0000 7fbda95a2700 1 -- 172.21.15.38:0/3376070132 >> 172.21.15.38:0/3376070132 conn(0x7fbda40fcdd0 msgr2=0x7fbda40fd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.088+0000 7fbda95a2700 1 -- 172.21.15.38:0/3376070132 shutdown_connections 2024-04-22T21:00:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:29.088+0000 7fbda95a2700 1 -- 172.21.15.38:0/3376070132 wait complete. 2024-04-22T21:00:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:mon is available 2024-04-22T21:00:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:Assimilating anything we can from ceph.conf... 2024-04-22T21:00:33.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:32 smithi038 ceph-mon[26004]: from='client.? 172.21.15.38:0/2525141629' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T21:00:33.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:32 smithi038 ceph-mon[26004]: from='client.? 172.21.15.38:0/2525141629' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-22T21:00:33.491 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T21:00:33.491 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T21:00:33.491 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout fsid = 3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T21:00:33.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.38:3300,v1:172.21.15.38:6789] 2024-04-22T21:00:33.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T21:00:33.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T21:00:33.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T21:00:33.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T21:00:33.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T21:00:33.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T21:00:33.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T21:00:33.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T21:00:33.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T21:00:33.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T21:00:33.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T21:00:33.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T21:00:33.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T21:00:33.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T21:00:33.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.840+0000 7f21ec32b700 1 Processor -- start 2024-04-22T21:00:33.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.840+0000 7f21ec32b700 1 -- start start 2024-04-22T21:00:33.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.840+0000 7f21ec32b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21e41018b0 0x7f21e4101cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:33.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.840+0000 7f21ec32b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e4102200 con 0x7f21e41018b0 2024-04-22T21:00:33.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.840+0000 7f21ea0c7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21e41018b0 0x7f21e4101cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:33.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.841+0000 7f21ea0c7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21e41018b0 0x7f21e4101cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54352/0 (socket says 172.21.15.38:54352) 2024-04-22T21:00:33.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.841+0000 7f21ea0c7700 1 -- 172.21.15.38:0/2854398045 learned_addr learned my addr 172.21.15.38:0/2854398045 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:33.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.841+0000 7f21ea0c7700 1 -- 172.21.15.38:0/2854398045 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e4102340 con 0x7f21e41018b0 2024-04-22T21:00:33.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.841+0000 7f21ea0c7700 1 --2- 172.21.15.38:0/2854398045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21e41018b0 0x7f21e4101cc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f21d400bb80 tx=0x7f21d400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce01defa01a7c611 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:33.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.841+0000 7f21e90c5700 1 -- 172.21.15.38:0/2854398045 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21d400dd90 con 0x7f21e41018b0 2024-04-22T21:00:33.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.841+0000 7f21e90c5700 1 -- 172.21.15.38:0/2854398045 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d4010470 con 0x7f21e41018b0 2024-04-22T21:00:33.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.841+0000 7f21e90c5700 1 -- 172.21.15.38:0/2854398045 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21d4010760 con 0x7f21e41018b0 2024-04-22T21:00:33.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.842+0000 7f21ec32b700 1 -- 172.21.15.38:0/2854398045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21e41018b0 msgr2=0x7f21e4101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:33.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.842+0000 7f21ec32b700 1 --2- 172.21.15.38:0/2854398045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21e41018b0 0x7f21e4101cc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f21d400bb80 tx=0x7f21d400b4f0 comp rx=0 tx=0).stop 2024-04-22T21:00:33.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.842+0000 7f21ec32b700 1 -- 172.21.15.38:0/2854398045 shutdown_connections 2024-04-22T21:00:33.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.842+0000 7f21ec32b700 1 --2- 172.21.15.38:0/2854398045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21e41018b0 0x7f21e4101cc0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:33.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.842+0000 7f21ec32b700 1 -- 172.21.15.38:0/2854398045 >> 172.21.15.38:0/2854398045 conn(0x7f21e40fd050 msgr2=0x7f21e40ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:33.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.842+0000 7f21ec32b700 1 -- 172.21.15.38:0/2854398045 shutdown_connections 2024-04-22T21:00:33.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.842+0000 7f21ec32b700 1 -- 172.21.15.38:0/2854398045 wait complete. 2024-04-22T21:00:33.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.842+0000 7f21ec32b700 1 Processor -- start 2024-04-22T21:00:33.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.843+0000 7f21ec32b700 1 -- start start 2024-04-22T21:00:33.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.843+0000 7f21ec32b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21e41018b0 0x7f21e410ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:33.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.843+0000 7f21ec32b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e410f290 con 0x7f21e41018b0 2024-04-22T21:00:33.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.843+0000 7f21ea0c7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21e41018b0 0x7f21e410ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:33.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.843+0000 7f21ea0c7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21e41018b0 0x7f21e410ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54356/0 (socket says 172.21.15.38:54356) 2024-04-22T21:00:33.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.843+0000 7f21ea0c7700 1 -- 172.21.15.38:0/2525141629 learned_addr learned my addr 172.21.15.38:0/2525141629 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:33.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.843+0000 7f21ea0c7700 1 -- 172.21.15.38:0/2525141629 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d400b790 con 0x7f21e41018b0 2024-04-22T21:00:33.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.843+0000 7f21ea0c7700 1 --2- 172.21.15.38:0/2525141629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21e41018b0 0x7f21e410ed50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f21d4000c00 tx=0x7f21d401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:33.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.843+0000 7f21daffd700 1 -- 172.21.15.38:0/2525141629 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21d4018910 con 0x7f21e41018b0 2024-04-22T21:00:33.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.843+0000 7f21daffd700 1 -- 172.21.15.38:0/2525141629 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d401d070 con 0x7f21e41018b0 2024-04-22T21:00:33.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.843+0000 7f21ec32b700 1 -- 172.21.15.38:0/2525141629 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e410f490 con 0x7f21e41018b0 2024-04-22T21:00:33.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.844+0000 7f21daffd700 1 -- 172.21.15.38:0/2525141629 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21d4018bd0 con 0x7f21e41018b0 2024-04-22T21:00:33.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.844+0000 7f21ec32b700 1 -- 172.21.15.38:0/2525141629 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e410ffb0 con 0x7f21e41018b0 2024-04-22T21:00:33.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.844+0000 7f21daffd700 1 -- 172.21.15.38:0/2525141629 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f21d402e460 con 0x7f21e41018b0 2024-04-22T21:00:33.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.844+0000 7f21daffd700 1 -- 172.21.15.38:0/2525141629 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f21d4014070 con 0x7f21e41018b0 2024-04-22T21:00:33.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.845+0000 7f21ec32b700 1 -- 172.21.15.38:0/2525141629 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e405f5f0 con 0x7f21e41018b0 2024-04-22T21:00:33.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.847+0000 7f21daffd700 1 -- 172.21.15.38:0/2525141629 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f21d401f050 con 0x7f21e41018b0 2024-04-22T21:00:33.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.916+0000 7f21ec32b700 1 -- 172.21.15.38:0/2525141629 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f21e4102c80 con 0x7f21e41018b0 2024-04-22T21:00:33.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.924+0000 7f21daffd700 1 -- 172.21.15.38:0/2525141629 <== mon.0 v2:172.21.15.38:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f21d4018e00 con 0x7f21e41018b0 2024-04-22T21:00:33.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.924+0000 7f21daffd700 1 -- 172.21.15.38:0/2525141629 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f21e4102c80 con 0x7f21e41018b0 2024-04-22T21:00:33.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.925+0000 7f21ec32b700 1 -- 172.21.15.38:0/2525141629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21e41018b0 msgr2=0x7f21e410ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:33.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.925+0000 7f21ec32b700 1 --2- 172.21.15.38:0/2525141629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21e41018b0 0x7f21e410ed50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f21d4000c00 tx=0x7f21d401a000 comp rx=0 tx=0).stop 2024-04-22T21:00:33.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.925+0000 7f21ec32b700 1 -- 172.21.15.38:0/2525141629 shutdown_connections 2024-04-22T21:00:33.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.925+0000 7f21ec32b700 1 --2- 172.21.15.38:0/2525141629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21e41018b0 0x7f21e410ed50 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:33.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.925+0000 7f21ec32b700 1 -- 172.21.15.38:0/2525141629 >> 172.21.15.38:0/2525141629 conn(0x7f21e40fd050 msgr2=0x7f21e4105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:33.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.925+0000 7f21ec32b700 1 -- 172.21.15.38:0/2525141629 shutdown_connections 2024-04-22T21:00:33.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:32.925+0000 7f21ec32b700 1 -- 172.21.15.38:0/2525141629 wait complete. 2024-04-22T21:00:33.496 INFO:teuthology.orchestra.run.smithi038.stdout:Generating new minimal ceph.conf... 2024-04-22T21:00:34.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:34 smithi038 ceph-mon[26004]: from='client.? 172.21.15.38:0/130036271' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:00:34.612 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.924+0000 7f27d7e21700 1 Processor -- start 2024-04-22T21:00:34.612 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.924+0000 7f27d7e21700 1 -- start start 2024-04-22T21:00:34.612 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.924+0000 7f27d7e21700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d0102930 0x7f27d0102d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:34.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.924+0000 7f27d7e21700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d0103280 con 0x7f27d0102930 2024-04-22T21:00:34.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.924+0000 7f27d5bbd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d0102930 0x7f27d0102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:34.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.924+0000 7f27d5bbd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d0102930 0x7f27d0102d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54366/0 (socket says 172.21.15.38:54366) 2024-04-22T21:00:34.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.924+0000 7f27d5bbd700 1 -- 172.21.15.38:0/1419374622 learned_addr learned my addr 172.21.15.38:0/1419374622 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:34.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.924+0000 7f27d5bbd700 1 -- 172.21.15.38:0/1419374622 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d01033c0 con 0x7f27d0102930 2024-04-22T21:00:34.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.925+0000 7f27d5bbd700 1 --2- 172.21.15.38:0/1419374622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d0102930 0x7f27d0102d40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f27c40098e0 tx=0x7f27c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5286a274c00e3012 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:34.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.925+0000 7f27d4bbb700 1 -- 172.21.15.38:0/1419374622 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27c400b8f0 con 0x7f27d0102930 2024-04-22T21:00:34.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.925+0000 7f27d4bbb700 1 -- 172.21.15.38:0/1419374622 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f27c4011420 con 0x7f27d0102930 2024-04-22T21:00:34.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.925+0000 7f27d4bbb700 1 -- 172.21.15.38:0/1419374622 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27c4010540 con 0x7f27d0102930 2024-04-22T21:00:34.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.925+0000 7f27d7e21700 1 -- 172.21.15.38:0/1419374622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d0102930 msgr2=0x7f27d0102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:34.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.925+0000 7f27d7e21700 1 --2- 172.21.15.38:0/1419374622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d0102930 0x7f27d0102d40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f27c40098e0 tx=0x7f27c4009250 comp rx=0 tx=0).stop 2024-04-22T21:00:34.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.925+0000 7f27d7e21700 1 -- 172.21.15.38:0/1419374622 shutdown_connections 2024-04-22T21:00:34.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.925+0000 7f27d7e21700 1 --2- 172.21.15.38:0/1419374622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d0102930 0x7f27d0102d40 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:34.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.925+0000 7f27d7e21700 1 -- 172.21.15.38:0/1419374622 >> 172.21.15.38:0/1419374622 conn(0x7f27d00fe140 msgr2=0x7f27d0100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:34.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.926+0000 7f27d7e21700 1 -- 172.21.15.38:0/1419374622 shutdown_connections 2024-04-22T21:00:34.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.926+0000 7f27d7e21700 1 -- 172.21.15.38:0/1419374622 wait complete. 2024-04-22T21:00:34.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.926+0000 7f27d7e21700 1 Processor -- start 2024-04-22T21:00:34.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.926+0000 7f27d7e21700 1 -- start start 2024-04-22T21:00:34.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.926+0000 7f27d7e21700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d0102930 0x7f27d0111750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:34.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.926+0000 7f27d7e21700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d0111c90 con 0x7f27d0102930 2024-04-22T21:00:34.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.926+0000 7f27d5bbd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d0102930 0x7f27d0111750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:34.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.926+0000 7f27d5bbd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d0102930 0x7f27d0111750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54382/0 (socket says 172.21.15.38:54382) 2024-04-22T21:00:34.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.926+0000 7f27d5bbd700 1 -- 172.21.15.38:0/130036271 learned_addr learned my addr 172.21.15.38:0/130036271 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:34.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.927+0000 7f27d5bbd700 1 -- 172.21.15.38:0/130036271 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c40094f0 con 0x7f27d0102930 2024-04-22T21:00:34.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.927+0000 7f27d5bbd700 1 --2- 172.21.15.38:0/130036271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d0102930 0x7f27d0111750 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f27c4011ef0 tx=0x7f27c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:34.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.927+0000 7f27c27fc700 1 -- 172.21.15.38:0/130036271 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27c4010910 con 0x7f27d0102930 2024-04-22T21:00:34.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.927+0000 7f27c27fc700 1 -- 172.21.15.38:0/130036271 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f27c4010f00 con 0x7f27d0102930 2024-04-22T21:00:34.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.927+0000 7f27d7e21700 1 -- 172.21.15.38:0/130036271 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d0111e90 con 0x7f27d0102930 2024-04-22T21:00:34.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.927+0000 7f27c27fc700 1 -- 172.21.15.38:0/130036271 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27c401ae50 con 0x7f27d0102930 2024-04-22T21:00:34.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.927+0000 7f27d7e21700 1 -- 172.21.15.38:0/130036271 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d0112a00 con 0x7f27d0102930 2024-04-22T21:00:34.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.927+0000 7f27c27fc700 1 -- 172.21.15.38:0/130036271 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f27c40244a0 con 0x7f27d0102930 2024-04-22T21:00:34.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.927+0000 7f27c27fc700 1 -- 172.21.15.38:0/130036271 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f27c402d740 con 0x7f27d0102930 2024-04-22T21:00:34.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.928+0000 7f27d7e21700 1 -- 172.21.15.38:0/130036271 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d005f5f0 con 0x7f27d0102930 2024-04-22T21:00:34.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:33.930+0000 7f27c27fc700 1 -- 172.21.15.38:0/130036271 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f27c4029070 con 0x7f27d0102930 2024-04-22T21:00:34.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:34.000+0000 7f27d7e21700 1 -- 172.21.15.38:0/130036271 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f27d0104620 con 0x7f27d0102930 2024-04-22T21:00:34.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:34.000+0000 7f27c27fc700 1 -- 172.21.15.38:0/130036271 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f27c401a430 con 0x7f27d0102930 2024-04-22T21:00:34.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:34.001+0000 7f27d7e21700 1 -- 172.21.15.38:0/130036271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d0102930 msgr2=0x7f27d0111750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:34.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:34.001+0000 7f27d7e21700 1 --2- 172.21.15.38:0/130036271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d0102930 0x7f27d0111750 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f27c4011ef0 tx=0x7f27c4009250 comp rx=0 tx=0).stop 2024-04-22T21:00:34.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:34.001+0000 7f27d7e21700 1 -- 172.21.15.38:0/130036271 shutdown_connections 2024-04-22T21:00:34.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:34.001+0000 7f27d7e21700 1 --2- 172.21.15.38:0/130036271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d0102930 0x7f27d0111750 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:34.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:34.001+0000 7f27d7e21700 1 -- 172.21.15.38:0/130036271 >> 172.21.15.38:0/130036271 conn(0x7f27d00fe140 msgr2=0x7f27d00feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:34.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:34.001+0000 7f27d7e21700 1 -- 172.21.15.38:0/130036271 shutdown_connections 2024-04-22T21:00:34.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:34.001+0000 7f27d7e21700 1 -- 172.21.15.38:0/130036271 wait complete. 2024-04-22T21:00:34.616 INFO:teuthology.orchestra.run.smithi038.stdout:Restarting the monitor... 2024-04-22T21:00:34.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:34 smithi038 systemd[1]: Stopping Ceph mon.smithi038 for 3c5f5352-00eb-11ef-bc93-c7b262605968... 2024-04-22T21:00:35.183 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:34 smithi038 ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-mon-smithi038[26000]: 2024-04-22T21:00:34.895+0000 7f0fbd078700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi038 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:00:35.183 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:34 smithi038 ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-mon-smithi038[26000]: 2024-04-22T21:00:34.895+0000 7f0fbd078700 -1 mon.smithi038@0(leader) e1 *** Got Signal Terminated *** 2024-04-22T21:00:35.183 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:34 smithi038 podman[26250]: 2024-04-22 21:00:34.967937265 +0000 UTC m=+0.180970060 container died 11af5584d9d0787d1f015e396c6d00bbf63d7dbc83b18a05076f037bbc302f93 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-mon-smithi038, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=quincy-70d0cb4, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-22T21:00:35.475 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:35 smithi038 podman[26250]: 2024-04-22 21:00:35.270697496 +0000 UTC m=+0.483730298 container cleanup 11af5584d9d0787d1f015e396c6d00bbf63d7dbc83b18a05076f037bbc302f93 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-mon-smithi038, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=quincy-70d0cb4, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-22T21:00:35.476 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:35 smithi038 bash[26250]: ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-mon-smithi038 2024-04-22T21:00:35.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:35 smithi038 podman[26264]: 2024-04-22 21:00:35.480226195 +0000 UTC m=+0.515486367 container remove 11af5584d9d0787d1f015e396c6d00bbf63d7dbc83b18a05076f037bbc302f93 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-mon-smithi038, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=quincy-70d0cb4, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-22T21:00:35.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:35 smithi038 systemd[1]: ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@mon.smithi038.service: Deactivated successfully. 2024-04-22T21:00:35.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:35 smithi038 systemd[1]: Stopped Ceph mon.smithi038 for 3c5f5352-00eb-11ef-bc93-c7b262605968. 2024-04-22T21:00:35.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:35 smithi038 systemd[1]: Starting Ceph mon.smithi038 for 3c5f5352-00eb-11ef-bc93-c7b262605968... 2024-04-22T21:00:36.124 INFO:teuthology.orchestra.run.smithi038.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-22T21:00:36.143 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:35 smithi038 podman[26354]: 2024-04-22 21:00:35.897595429 +0000 UTC m=+0.093458898 container create 2137696f8742688209494816111b6028cb21926ce7e451eb9700e298584d9683 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-mon-smithi038, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-70d0cb4, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-22T21:00:36.143 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:35 smithi038 podman[26354]: 2024-04-22 21:00:35.822398604 +0000 UTC m=+0.018262090 image pull cafd8ddac8fa176bf00aeacab8beaeeffcb0c5261d2d415c33683cb8bed0534e quay.ceph.io/ceph-ci/ceph:quincy 2024-04-22T21:00:36.143 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 podman[26354]: 2024-04-22 21:00:36.049893138 +0000 UTC m=+0.245756611 container init 2137696f8742688209494816111b6028cb21926ce7e451eb9700e298584d9683 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-mon-smithi038, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-70d0cb4, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-22T21:00:36.144 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 podman[26354]: 2024-04-22 21:00:36.052797198 +0000 UTC m=+0.248660669 container start 2137696f8742688209494816111b6028cb21926ce7e451eb9700e298584d9683 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-mon-smithi038, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=quincy-70d0cb4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-22T21:00:36.144 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T21:00:36.144 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: ceph version 17.2.7-884-g70d0cb4b (70d0cb4b9d576828afb0a05dcf8732894d343ece) quincy (stable), process ceph-mon, pid 2 2024-04-22T21:00:36.144 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: pidfile_write: ignore empty --pid-file 2024-04-22T21:00:36.144 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: load: jerasure load: lrc 2024-04-22T21:00:36.144 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: RocksDB version: 6.15.5 2024-04-22T21:00:36.144 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-22T21:00:36.144 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Compile date Apr 22 2024 2024-04-22T21:00:36.144 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: DB SUMMARY 2024-04-22T21:00:36.144 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: DB Session ID: 2179M1PEG9XYP63MUT0F 2024-04-22T21:00:36.144 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: CURRENT file: CURRENT 2024-04-22T21:00:36.144 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T21:00:36.144 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-22T21:00:36.145 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi038/store.db dir, Total Num: 1, files: 000008.sst 2024-04-22T21:00:36.145 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi038/store.db: 000010.log size: 98462 ; 2024-04-22T21:00:36.145 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.error_if_exists: 0 2024-04-22T21:00:36.145 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.create_if_missing: 0 2024-04-22T21:00:36.145 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.paranoid_checks: 1 2024-04-22T21:00:36.146 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T21:00:36.146 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.env: 0x557e1d62bb00 2024-04-22T21:00:36.146 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.fs: Posix File System 2024-04-22T21:00:36.147 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.info_log: 0x557e1ea8dc00 2024-04-22T21:00:36.147 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T21:00:36.147 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.statistics: (nil) 2024-04-22T21:00:36.147 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.use_fsync: 0 2024-04-22T21:00:36.147 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_log_file_size: 0 2024-04-22T21:00:36.147 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T21:00:36.147 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T21:00:36.147 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T21:00:36.147 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T21:00:36.147 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.allow_fallocate: 1 2024-04-22T21:00:36.147 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T21:00:36.147 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T21:00:36.147 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.use_direct_reads: 0 2024-04-22T21:00:36.148 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T21:00:36.148 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T21:00:36.148 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.db_log_dir: 2024-04-22T21:00:36.148 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi038/store.db 2024-04-22T21:00:36.148 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T21:00:36.148 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T21:00:36.148 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T21:00:36.148 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T21:00:36.148 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T21:00:36.148 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T21:00:36.148 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T21:00:36.148 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T21:00:36.148 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.write_buffer_manager: 0x557e1f802a20 2024-04-22T21:00:36.148 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T21:00:36.149 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-22T21:00:36.149 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T21:00:36.149 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T21:00:36.149 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T21:00:36.149 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T21:00:36.149 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T21:00:36.149 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T21:00:36.149 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T21:00:36.149 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.unordered_write: 0 2024-04-22T21:00:36.149 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T21:00:36.149 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T21:00:36.149 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T21:00:36.149 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T21:00:36.149 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.row_cache: None 2024-04-22T21:00:36.150 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.wal_filter: None 2024-04-22T21:00:36.150 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T21:00:36.150 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T21:00:36.150 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.preserve_deletes: 0 2024-04-22T21:00:36.150 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.two_write_queues: 0 2024-04-22T21:00:36.150 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T21:00:36.150 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.atomic_flush: 0 2024-04-22T21:00:36.150 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T21:00:36.150 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T21:00:36.150 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T21:00:36.150 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.log_readahead_size: 0 2024-04-22T21:00:36.150 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T21:00:36.150 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T21:00:36.150 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T21:00:36.151 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T21:00:36.151 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T21:00:36.151 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T21:00:36.151 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_background_jobs: 2 2024-04-22T21:00:36.151 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_background_compactions: -1 2024-04-22T21:00:36.151 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_subcompactions: 1 2024-04-22T21:00:36.151 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T21:00:36.151 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T21:00:36.151 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T21:00:36.151 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T21:00:36.151 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T21:00:36.151 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T21:00:36.151 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T21:00:36.151 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T21:00:36.152 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_open_files: -1 2024-04-22T21:00:36.152 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T21:00:36.152 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T21:00:36.152 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T21:00:36.152 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T21:00:36.152 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_background_flushes: -1 2024-04-22T21:00:36.152 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Compression algorithms supported: 2024-04-22T21:00:36.152 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T21:00:36.152 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: kZSTD supported: 0 2024-04-22T21:00:36.152 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: kXpressCompression supported: 0 2024-04-22T21:00:36.152 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T21:00:36.152 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: kZlibCompression supported: 1 2024-04-22T21:00:36.152 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: kSnappyCompression supported: 1 2024-04-22T21:00:36.152 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: kLZ4Compression supported: 1 2024-04-22T21:00:36.152 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: kBZip2Compression supported: 0 2024-04-22T21:00:36.153 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T21:00:36.153 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi038/store.db/MANIFEST-000009 2024-04-22T21:00:36.153 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-04-22T21:00:36.153 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T21:00:36.153 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.merge_operator: 2024-04-22T21:00:36.153 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compaction_filter: None 2024-04-22T21:00:36.153 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T21:00:36.153 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T21:00:36.153 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T21:00:36.153 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T21:00:36.154 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557e1eae2f58) 2024-04-22T21:00:36.154 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: cache_index_and_filter_blocks: 1 2024-04-22T21:00:36.154 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T21:00:36.154 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T21:00:36.154 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: pin_top_level_index_and_filter: 1 2024-04-22T21:00:36.154 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: index_type: 0 2024-04-22T21:00:36.154 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: data_block_index_type: 0 2024-04-22T21:00:36.154 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: index_shortening: 1 2024-04-22T21:00:36.154 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T21:00:36.154 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: hash_index_allow_collision: 1 2024-04-22T21:00:36.154 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: checksum: 1 2024-04-22T21:00:36.154 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: no_block_cache: 0 2024-04-22T21:00:36.154 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: block_cache: 0x557e1eb15090 2024-04-22T21:00:36.155 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: block_cache_name: BinnedLRUCache 2024-04-22T21:00:36.155 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: block_cache_options: 2024-04-22T21:00:36.155 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: capacity : 536870912 2024-04-22T21:00:36.155 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: num_shard_bits : 4 2024-04-22T21:00:36.155 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: strict_capacity_limit : 0 2024-04-22T21:00:36.155 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: high_pri_pool_ratio: 0.000 2024-04-22T21:00:36.155 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: block_cache_compressed: (nil) 2024-04-22T21:00:36.155 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: persistent_cache: (nil) 2024-04-22T21:00:36.155 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: block_size: 4096 2024-04-22T21:00:36.155 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: block_size_deviation: 10 2024-04-22T21:00:36.155 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: block_restart_interval: 16 2024-04-22T21:00:36.155 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: index_block_restart_interval: 1 2024-04-22T21:00:36.155 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: metadata_block_size: 4096 2024-04-22T21:00:36.155 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: partition_filters: 0 2024-04-22T21:00:36.156 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: use_delta_encoding: 1 2024-04-22T21:00:36.156 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-22T21:00:36.156 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: whole_key_filtering: 1 2024-04-22T21:00:36.156 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: verify_compression: 0 2024-04-22T21:00:36.156 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: read_amp_bytes_per_bit: 0 2024-04-22T21:00:36.156 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: format_version: 4 2024-04-22T21:00:36.156 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: enable_index_compression: 1 2024-04-22T21:00:36.156 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: block_align: 0 2024-04-22T21:00:36.156 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T21:00:36.156 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T21:00:36.156 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compression: NoCompression 2024-04-22T21:00:36.156 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T21:00:36.156 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T21:00:36.156 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T21:00:36.156 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.num_levels: 7 2024-04-22T21:00:36.157 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T21:00:36.157 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T21:00:36.157 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T21:00:36.157 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T21:00:36.157 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T21:00:36.157 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T21:00:36.157 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T21:00:36.157 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T21:00:36.157 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T21:00:36.157 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T21:00:36.157 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T21:00:36.157 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T21:00:36.157 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T21:00:36.157 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T21:00:36.158 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T21:00:36.158 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T21:00:36.158 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T21:00:36.158 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T21:00:36.158 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T21:00:36.158 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T21:00:36.158 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T21:00:36.158 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T21:00:36.158 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T21:00:36.158 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T21:00:36.158 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T21:00:36.158 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T21:00:36.158 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T21:00:36.158 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T21:00:36.159 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T21:00:36.159 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T21:00:36.159 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T21:00:36.159 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T21:00:36.159 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T21:00:36.159 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T21:00:36.159 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.arena_block_size: 4194304 2024-04-22T21:00:36.159 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T21:00:36.159 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T21:00:36.159 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-22T21:00:36.159 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T21:00:36.159 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T21:00:36.159 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T21:00:36.159 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T21:00:36.159 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T21:00:36.159 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T21:00:36.160 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T21:00:36.160 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T21:00:36.160 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T21:00:36.160 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T21:00:36.160 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T21:00:36.160 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.table_properties_collectors: 2024-04-22T21:00:36.160 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.inplace_update_support: 0 2024-04-22T21:00:36.160 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T21:00:36.160 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T21:00:36.160 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T21:00:36.160 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T21:00:36.160 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.bloom_locality: 0 2024-04-22T21:00:36.160 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.max_successive_merges: 0 2024-04-22T21:00:36.160 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T21:00:36.161 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T21:00:36.161 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T21:00:36.161 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T21:00:36.161 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.ttl: 2592000 2024-04-22T21:00:36.161 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T21:00:36.161 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.enable_blob_files: false 2024-04-22T21:00:36.161 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.min_blob_size: 0 2024-04-22T21:00:36.161 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T21:00:36.161 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T21:00:36.161 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T21:00:36.161 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T21:00:36.161 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi038/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T21:00:36.161 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-04-22T21:00:36.161 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-04-22T21:00:36.161 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713819636084869, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-22T21:00:36.162 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-04-22T21:00:36.162 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-22T21:00:36.162 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713819636085800, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 93513, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 91625, "index_size": 248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13606, "raw_average_key_size": 49, "raw_value_size": 84163, "raw_average_value_size": 306, "num_data_blocks": 11, "num_entries": 275, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713819636, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "e48e2838-d03d-48a9-8c73-2fd6a6780bca", "db_session_id": "2179M1PEG9XYP63MUT0F"}} 2024-04-22T21:00:36.162 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-04-22T21:00:36.162 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713819636086248, "job": 1, "event": "recovery_finished"} 2024-04-22T21:00:36.162 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi038/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T21:00:36.162 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x557e1eaa9180 2024-04-22T21:00:36.162 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: DB pointer 0x557e1eb7e000 2024-04-22T21:00:36.162 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-04-22T21:00:36.162 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-22T21:00:36.162 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: ** DB Stats ** 2024-04-22T21:00:36.162 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:00:36.162 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T21:00:36.162 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T21:00:36.163 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T21:00:36.163 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T21:00:36.163 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-22T21:00:36.163 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T21:00:36.163 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: 2024-04-22T21:00:36.163 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: ** Compaction Stats [default] ** 2024-04-22T21:00:36.163 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:00:36.163 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:00:36.163 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: L0 2/0 93.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.5 0.00 0.00 1 0.001 0 0 2024-04-22T21:00:36.163 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Sum 2/0 93.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.5 0.00 0.00 1 0.001 0 0 2024-04-22T21:00:36.163 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.5 0.00 0.00 1 0.001 0 0 2024-04-22T21:00:36.163 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: 2024-04-22T21:00:36.163 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: ** Compaction Stats [default] ** 2024-04-22T21:00:36.163 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:00:36.164 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:00:36.164 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.5 0.00 0.00 1 0.001 0 0 2024-04-22T21:00:36.164 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:00:36.164 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T21:00:36.164 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T21:00:36.164 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T21:00:36.164 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T21:00:36.164 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T21:00:36.164 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Cumulative compaction: 0.00 GB write, 10.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:00:36.164 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Interval compaction: 0.00 GB write, 10.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:00:36.164 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T21:00:36.164 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: 2024-04-22T21:00:36.164 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T21:00:36.164 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: 2024-04-22T21:00:36.164 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: ** Compaction Stats [default] ** 2024-04-22T21:00:36.165 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:00:36.165 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:00:36.165 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: L0 2/0 93.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.5 0.00 0.00 1 0.001 0 0 2024-04-22T21:00:36.165 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Sum 2/0 93.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.5 0.00 0.00 1 0.001 0 0 2024-04-22T21:00:36.165 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-22T21:00:36.165 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: 2024-04-22T21:00:36.165 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: ** Compaction Stats [default] ** 2024-04-22T21:00:36.165 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:00:36.165 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:00:36.165 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.5 0.00 0.00 1 0.001 0 0 2024-04-22T21:00:36.165 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:00:36.165 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T21:00:36.165 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T21:00:36.165 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T21:00:36.165 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T21:00:36.166 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T21:00:36.166 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Cumulative compaction: 0.00 GB write, 10.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:00:36.166 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:00:36.166 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T21:00:36.166 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: 2024-04-22T21:00:36.166 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T21:00:36.166 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: starting mon.smithi038 rank 0 at public addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] at bind addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi038 fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T21:00:36.166 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: mon.smithi038@-1(???) e1 preinit fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T21:00:36.166 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: mon.smithi038@-1(???).mds e1 new map 2024-04-22T21:00:36.166 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: mon.smithi038@-1(???).mds e1 print_map 2024-04-22T21:00:36.166 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: e1 2024-04-22T21:00:36.166 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T21:00:36.166 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T21:00:36.166 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: legacy client fscid: -1 2024-04-22T21:00:36.167 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: 2024-04-22T21:00:36.167 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout: No filesystems configured 2024-04-22T21:00:36.167 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: mon.smithi038@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T21:00:36.167 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: mon.smithi038@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:00:36.167 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: mon.smithi038@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:00:36.167 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: mon.smithi038@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:00:36.167 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: mon.smithi038@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-22T21:00:36.167 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 bash[26354]: 2137696f8742688209494816111b6028cb21926ce7e451eb9700e298584d9683 2024-04-22T21:00:36.167 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 systemd[1]: Started Ceph mon.smithi038 for 3c5f5352-00eb-11ef-bc93-c7b262605968. 2024-04-22T21:00:36.167 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: mon.smithi038 is new leader, mons smithi038 in quorum (ranks 0) 2024-04-22T21:00:36.167 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: monmap e1: 1 mons at {smithi038=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-22T21:00:36.167 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: fsmap 2024-04-22T21:00:36.167 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T21:00:36.167 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:36 smithi038 ceph-mon[26374]: mgrmap e1: no daemons active 2024-04-22T21:00:37.133 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.529+0000 7ffb9c41b700 1 Processor -- start 2024-04-22T21:00:37.133 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.529+0000 7ffb9c41b700 1 -- start start 2024-04-22T21:00:37.133 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.529+0000 7ffb9c41b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb94078b80 0x7ffb94077220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:37.133 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.529+0000 7ffb9c41b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb94078f90 con 0x7ffb94078b80 2024-04-22T21:00:37.133 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.530+0000 7ffb9a1b7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb94078b80 0x7ffb94077220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:37.133 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.530+0000 7ffb9a1b7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb94078b80 0x7ffb94077220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54396/0 (socket says 172.21.15.38:54396) 2024-04-22T21:00:37.133 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.530+0000 7ffb9a1b7700 1 -- 172.21.15.38:0/3699654505 learned_addr learned my addr 172.21.15.38:0/3699654505 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:37.133 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.530+0000 7ffb9a1b7700 1 -- 172.21.15.38:0/3699654505 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb940790d0 con 0x7ffb94078b80 2024-04-22T21:00:37.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.530+0000 7ffb9a1b7700 1 --2- 172.21.15.38:0/3699654505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb94078b80 0x7ffb94077220 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009e50 tx=0x7ffb8c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0544d3edbe8f765 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:37.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.530+0000 7ffb991b5700 1 -- 172.21.15.38:0/3699654505 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb8c0092e0 con 0x7ffb94078b80 2024-04-22T21:00:37.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.530+0000 7ffb991b5700 1 -- 172.21.15.38:0/3699654505 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ffb8c012430 con 0x7ffb94078b80 2024-04-22T21:00:37.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.531+0000 7ffb991b5700 1 -- 172.21.15.38:0/3699654505 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb8c0075c0 con 0x7ffb94078b80 2024-04-22T21:00:37.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.531+0000 7ffb9c41b700 1 -- 172.21.15.38:0/3699654505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb94078b80 msgr2=0x7ffb94077220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:37.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.531+0000 7ffb9c41b700 1 --2- 172.21.15.38:0/3699654505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb94078b80 0x7ffb94077220 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009e50 tx=0x7ffb8c009840 comp rx=0 tx=0).stop 2024-04-22T21:00:37.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.531+0000 7ffb9c41b700 1 -- 172.21.15.38:0/3699654505 shutdown_connections 2024-04-22T21:00:37.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.531+0000 7ffb9c41b700 1 --2- 172.21.15.38:0/3699654505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb94078b80 0x7ffb94077220 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:37.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.531+0000 7ffb9c41b700 1 -- 172.21.15.38:0/3699654505 >> 172.21.15.38:0/3699654505 conn(0x7ffb940fe0e0 msgr2=0x7ffb941004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:37.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.531+0000 7ffb9c41b700 1 -- 172.21.15.38:0/3699654505 shutdown_connections 2024-04-22T21:00:37.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.531+0000 7ffb9c41b700 1 -- 172.21.15.38:0/3699654505 wait complete. 2024-04-22T21:00:37.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.532+0000 7ffb9c41b700 1 Processor -- start 2024-04-22T21:00:37.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.532+0000 7ffb9c41b700 1 -- start start 2024-04-22T21:00:37.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.532+0000 7ffb9c41b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb94078b80 0x7ffb94113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:37.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.532+0000 7ffb9c41b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb94113e50 con 0x7ffb94078b80 2024-04-22T21:00:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.532+0000 7ffb9a1b7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb94078b80 0x7ffb94113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.532+0000 7ffb9a1b7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb94078b80 0x7ffb94113910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54400/0 (socket says 172.21.15.38:54400) 2024-04-22T21:00:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.532+0000 7ffb9a1b7700 1 -- 172.21.15.38:0/3857801620 learned_addr learned my addr 172.21.15.38:0/3857801620 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.532+0000 7ffb9a1b7700 1 -- 172.21.15.38:0/3857801620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8c009ae0 con 0x7ffb94078b80 2024-04-22T21:00:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.532+0000 7ffb9a1b7700 1 --2- 172.21.15.38:0/3857801620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb94078b80 0x7ffb94113910 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009fd0 tx=0x7ffb8c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.533+0000 7ffb92ffd700 1 -- 172.21.15.38:0/3857801620 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb8c0092e0 con 0x7ffb94078b80 2024-04-22T21:00:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.533+0000 7ffb92ffd700 1 -- 172.21.15.38:0/3857801620 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ffb8c007cc0 con 0x7ffb94078b80 2024-04-22T21:00:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.533+0000 7ffb92ffd700 1 -- 172.21.15.38:0/3857801620 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb8c01c970 con 0x7ffb94078b80 2024-04-22T21:00:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.533+0000 7ffb9c41b700 1 -- 172.21.15.38:0/3857801620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb94114050 con 0x7ffb94078b80 2024-04-22T21:00:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.533+0000 7ffb9c41b700 1 -- 172.21.15.38:0/3857801620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb94114bc0 con 0x7ffb94078b80 2024-04-22T21:00:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.533+0000 7ffb92ffd700 1 -- 172.21.15.38:0/3857801620 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7ffb8c019070 con 0x7ffb94078b80 2024-04-22T21:00:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.533+0000 7ffb92ffd700 1 -- 172.21.15.38:0/3857801620 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ffb8c025b40 con 0x7ffb94078b80 2024-04-22T21:00:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.534+0000 7ffb9c41b700 1 -- 172.21.15.38:0/3857801620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb9405f5f0 con 0x7ffb94078b80 2024-04-22T21:00:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.536+0000 7ffb92ffd700 1 -- 172.21.15.38:0/3857801620 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7ffb8c02a060 con 0x7ffb94078b80 2024-04-22T21:00:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.608+0000 7ffb9c41b700 1 -- 172.21.15.38:0/3857801620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7ffb94108c20 con 0x7ffb94078b80 2024-04-22T21:00:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.612+0000 7ffb92ffd700 1 -- 172.21.15.38:0/3857801620 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7ffb94108c20 con 0x7ffb94078b80 2024-04-22T21:00:37.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.612+0000 7ffb9c41b700 1 -- 172.21.15.38:0/3857801620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb94078b80 msgr2=0x7ffb94113910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:37.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.612+0000 7ffb9c41b700 1 --2- 172.21.15.38:0/3857801620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb94078b80 0x7ffb94113910 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009fd0 tx=0x7ffb8c009840 comp rx=0 tx=0).stop 2024-04-22T21:00:37.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.612+0000 7ffb9c41b700 1 -- 172.21.15.38:0/3857801620 shutdown_connections 2024-04-22T21:00:37.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.612+0000 7ffb9c41b700 1 --2- 172.21.15.38:0/3857801620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb94078b80 0x7ffb94113910 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:37.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.612+0000 7ffb9c41b700 1 -- 172.21.15.38:0/3857801620 >> 172.21.15.38:0/3857801620 conn(0x7ffb940fe0e0 msgr2=0x7ffb940fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:37.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.613+0000 7ffb9c41b700 1 -- 172.21.15.38:0/3857801620 shutdown_connections 2024-04-22T21:00:37.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:36.613+0000 7ffb9c41b700 1 -- 172.21.15.38:0/3857801620 wait complete. 2024-04-22T21:00:37.136 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-22T21:00:37.136 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:00:37.136 INFO:teuthology.orchestra.run.smithi038.stdout:Creating mgr... 2024-04-22T21:00:37.136 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying port 9283 ... 2024-04-22T21:00:37.320 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@mgr.smithi038.lxfgut 2024-04-22T21:00:37.320 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@mgr.smithi038.lxfgut.service: Unit ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@mgr.smithi038.lxfgut.service not loaded. 2024-04-22T21:00:37.524 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3c5f5352-00eb-11ef-bc93-c7b262605968.target.wants/ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@mgr.smithi038.lxfgut.service → /etc/systemd/system/ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@.service. 2024-04-22T21:00:37.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:37 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3857801620' entity='client.admin' 2024-04-22T21:00:39.560 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T21:00:39.560 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-04-22T21:00:39.567 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T21:00:39.567 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-22T21:00:39.568 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-04-22T21:00:39.568 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T21:00:39.579 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T21:00:39.579 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-04-22T21:00:39.588 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T21:00:39.588 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-22T21:00:39.588 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-04-22T21:00:39.588 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-22T21:00:39.588 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr to start... 2024-04-22T21:00:39.588 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr... 2024-04-22T21:00:40.476 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:40 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/2599678137' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T21:00:40.757 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T21:00:40.757 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-22T21:00:40.757 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "3c5f5352-00eb-11ef-bc93-c7b262605968", 2024-04-22T21:00:40.757 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T21:00:40.757 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T21:00:40.757 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T21:00:40.757 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T21:00:40.757 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:40.757 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T21:00:40.757 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T21:00:40.757 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-04-22T21:00:40.757 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T21:00:40.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T21:00:40.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "smithi038" 2024-04-22T21:00:40.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T21:00:40.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-04-22T21:00:40.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T21:00:40.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T21:00:40.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-22T21:00:40.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T21:00:40.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:40.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T21:00:40.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T21:00:40.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T21:00:40.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T21:00:40.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T21:00:40.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T21:00:40.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T21:00:40.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T21:00:40.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:40.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T21:00:40.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T21:00:40.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T21:00:40.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T21:00:40.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T21:00:40.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T21:00:40.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T21:00:40.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T21:00:40.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T21:00:40.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:40.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T21:00:40.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T21:00:40.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T21:00:40.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T21:00:40.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:40.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T21:00:40.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T21:00:40.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T21:00:40.766 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T21:00:40.766 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T21:00:40.766 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T21:00:40.766 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T21:00:40.766 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T21:00:40.766 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T21:00:40.766 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:40.766 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T21:00:40.766 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T21:00:40.766 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T21:00:25.280583+0000", 2024-04-22T21:00:40.766 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T21:00:40.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:40.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T21:00:40.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-22T21:00:40.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.095+0000 7fab35339700 1 Processor -- start 2024-04-22T21:00:40.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.095+0000 7fab35339700 1 -- start start 2024-04-22T21:00:40.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.095+0000 7fab35339700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab301029a0 0x7fab30102db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:40.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.095+0000 7fab35339700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab301032f0 con 0x7fab301029a0 2024-04-22T21:00:40.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.095+0000 7fab2effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab301029a0 0x7fab30102db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:40.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.096+0000 7fab2effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab301029a0 0x7fab30102db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54420/0 (socket says 172.21.15.38:54420) 2024-04-22T21:00:40.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.096+0000 7fab2effd700 1 -- 172.21.15.38:0/630587309 learned_addr learned my addr 172.21.15.38:0/630587309 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:40.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.096+0000 7fab2effd700 1 -- 172.21.15.38:0/630587309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab30103430 con 0x7fab301029a0 2024-04-22T21:00:40.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.096+0000 7fab2effd700 1 --2- 172.21.15.38:0/630587309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab301029a0 0x7fab30102db0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fab200098e0 tx=0x7fab20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d14bab3e3c1d46 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:40.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.096+0000 7fab2dffb700 1 -- 172.21.15.38:0/630587309 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab2000b8f0 con 0x7fab301029a0 2024-04-22T21:00:40.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.096+0000 7fab2dffb700 1 -- 172.21.15.38:0/630587309 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fab20011420 con 0x7fab301029a0 2024-04-22T21:00:40.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.096+0000 7fab2dffb700 1 -- 172.21.15.38:0/630587309 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab20010540 con 0x7fab301029a0 2024-04-22T21:00:40.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.097+0000 7fab35339700 1 -- 172.21.15.38:0/630587309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab301029a0 msgr2=0x7fab30102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:40.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.097+0000 7fab35339700 1 --2- 172.21.15.38:0/630587309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab301029a0 0x7fab30102db0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fab200098e0 tx=0x7fab20009250 comp rx=0 tx=0).stop 2024-04-22T21:00:40.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.097+0000 7fab35339700 1 -- 172.21.15.38:0/630587309 shutdown_connections 2024-04-22T21:00:40.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.097+0000 7fab35339700 1 --2- 172.21.15.38:0/630587309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab301029a0 0x7fab30102db0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:40.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.097+0000 7fab35339700 1 -- 172.21.15.38:0/630587309 >> 172.21.15.38:0/630587309 conn(0x7fab300fe160 msgr2=0x7fab30100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:40.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.097+0000 7fab35339700 1 -- 172.21.15.38:0/630587309 shutdown_connections 2024-04-22T21:00:40.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.097+0000 7fab35339700 1 -- 172.21.15.38:0/630587309 wait complete. 2024-04-22T21:00:40.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.098+0000 7fab35339700 1 Processor -- start 2024-04-22T21:00:40.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.098+0000 7fab35339700 1 -- start start 2024-04-22T21:00:40.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.098+0000 7fab35339700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab301029a0 0x7fab30111720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:40.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.098+0000 7fab35339700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab30111c60 con 0x7fab301029a0 2024-04-22T21:00:40.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.098+0000 7fab2effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab301029a0 0x7fab30111720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:40.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.098+0000 7fab2effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab301029a0 0x7fab30111720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54426/0 (socket says 172.21.15.38:54426) 2024-04-22T21:00:40.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.098+0000 7fab2effd700 1 -- 172.21.15.38:0/2599678137 learned_addr learned my addr 172.21.15.38:0/2599678137 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:40.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.098+0000 7fab2effd700 1 -- 172.21.15.38:0/2599678137 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab200094f0 con 0x7fab301029a0 2024-04-22T21:00:40.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.099+0000 7fab2effd700 1 --2- 172.21.15.38:0/2599678137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab301029a0 0x7fab30111720 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fab20010760 tx=0x7fab20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:40.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.099+0000 7fab177fe700 1 -- 172.21.15.38:0/2599678137 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab20010800 con 0x7fab301029a0 2024-04-22T21:00:40.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.099+0000 7fab177fe700 1 -- 172.21.15.38:0/2599678137 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fab20010df0 con 0x7fab301029a0 2024-04-22T21:00:40.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.099+0000 7fab35339700 1 -- 172.21.15.38:0/2599678137 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab30111e60 con 0x7fab301029a0 2024-04-22T21:00:40.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.099+0000 7fab177fe700 1 -- 172.21.15.38:0/2599678137 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab200193c0 con 0x7fab301029a0 2024-04-22T21:00:40.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.099+0000 7fab35339700 1 -- 172.21.15.38:0/2599678137 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab301129d0 con 0x7fab301029a0 2024-04-22T21:00:40.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.100+0000 7fab177fe700 1 -- 172.21.15.38:0/2599678137 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fab20019680 con 0x7fab301029a0 2024-04-22T21:00:40.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.100+0000 7fab177fe700 1 -- 172.21.15.38:0/2599678137 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fab20024790 con 0x7fab301029a0 2024-04-22T21:00:40.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.100+0000 7fab35339700 1 -- 172.21.15.38:0/2599678137 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3005f5f0 con 0x7fab301029a0 2024-04-22T21:00:40.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.102+0000 7fab177fe700 1 -- 172.21.15.38:0/2599678137 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fab2001a6c0 con 0x7fab301029a0 2024-04-22T21:00:40.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.177+0000 7fab35339700 1 -- 172.21.15.38:0/2599678137 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fab3004bed0 con 0x7fab301029a0 2024-04-22T21:00:40.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.178+0000 7fab177fe700 1 -- 172.21.15.38:0/2599678137 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fab2001a430 con 0x7fab301029a0 2024-04-22T21:00:40.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.179+0000 7fab35339700 1 -- 172.21.15.38:0/2599678137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab301029a0 msgr2=0x7fab30111720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:40.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.179+0000 7fab35339700 1 --2- 172.21.15.38:0/2599678137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab301029a0 0x7fab30111720 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fab20010760 tx=0x7fab20009250 comp rx=0 tx=0).stop 2024-04-22T21:00:40.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.179+0000 7fab35339700 1 -- 172.21.15.38:0/2599678137 shutdown_connections 2024-04-22T21:00:40.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.179+0000 7fab35339700 1 --2- 172.21.15.38:0/2599678137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab301029a0 0x7fab30111720 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:40.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.179+0000 7fab35339700 1 -- 172.21.15.38:0/2599678137 >> 172.21.15.38:0/2599678137 conn(0x7fab300fe160 msgr2=0x7fab300fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:40.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.179+0000 7fab35339700 1 -- 172.21.15.38:0/2599678137 shutdown_connections 2024-04-22T21:00:40.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:40.179+0000 7fab35339700 1 -- 172.21.15.38:0/2599678137 wait complete. 2024-04-22T21:00:40.770 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (1/15)... 2024-04-22T21:00:43.607 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:43 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/2863467626' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T21:00:43.928 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T21:00:43.929 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-22T21:00:43.929 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "3c5f5352-00eb-11ef-bc93-c7b262605968", 2024-04-22T21:00:43.929 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T21:00:43.929 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T21:00:43.929 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T21:00:43.929 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T21:00:43.929 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:43.929 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T21:00:43.929 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T21:00:43.929 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-04-22T21:00:43.929 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T21:00:43.929 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T21:00:43.929 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "smithi038" 2024-04-22T21:00:43.930 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T21:00:43.930 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-22T21:00:43.930 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T21:00:43.930 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T21:00:43.930 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-22T21:00:43.930 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T21:00:43.930 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:43.930 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T21:00:43.930 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T21:00:43.930 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T21:00:43.930 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T21:00:43.930 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T21:00:43.930 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T21:00:43.930 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T21:00:43.931 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T21:00:43.931 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:43.931 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T21:00:43.931 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T21:00:43.931 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T21:00:43.931 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T21:00:43.931 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T21:00:43.931 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T21:00:43.931 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T21:00:43.931 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T21:00:43.931 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T21:00:43.931 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:43.931 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T21:00:43.931 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T21:00:43.931 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T21:00:43.932 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T21:00:43.932 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:43.932 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T21:00:43.932 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T21:00:43.932 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T21:00:43.932 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T21:00:43.932 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T21:00:43.932 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T21:00:43.932 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T21:00:43.932 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T21:00:43.932 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T21:00:43.932 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:43.932 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T21:00:43.932 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T21:00:43.932 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T21:00:25.280583+0000", 2024-04-22T21:00:43.933 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T21:00:43.933 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:43.933 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T21:00:43.933 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-22T21:00:43.933 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.216+0000 7f9f5d6c0700 1 Processor -- start 2024-04-22T21:00:43.933 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.216+0000 7f9f5d6c0700 1 -- start start 2024-04-22T21:00:43.933 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.216+0000 7f9f5d6c0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f58101820 0x7f9f58101c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:43.933 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.216+0000 7f9f5d6c0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f58102170 con 0x7f9f58101820 2024-04-22T21:00:43.933 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.216+0000 7f9f56ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f58101820 0x7f9f58101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:43.933 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.216+0000 7f9f56ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f58101820 0x7f9f58101c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49982/0 (socket says 172.21.15.38:49982) 2024-04-22T21:00:43.933 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.216+0000 7f9f56ffd700 1 -- 172.21.15.38:0/1739411987 learned_addr learned my addr 172.21.15.38:0/1739411987 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:43.933 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.217+0000 7f9f56ffd700 1 -- 172.21.15.38:0/1739411987 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f581022b0 con 0x7f9f58101820 2024-04-22T21:00:43.933 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.217+0000 7f9f56ffd700 1 --2- 172.21.15.38:0/1739411987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f58101820 0x7f9f58101c30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9f480098e0 tx=0x7f9f48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab6a666cff442d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:43.933 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.217+0000 7f9f55ffb700 1 -- 172.21.15.38:0/1739411987 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f4800b860 con 0x7f9f58101820 2024-04-22T21:00:43.933 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.217+0000 7f9f55ffb700 1 -- 172.21.15.38:0/1739411987 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9f4800be50 con 0x7f9f58101820 2024-04-22T21:00:43.934 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.217+0000 7f9f55ffb700 1 -- 172.21.15.38:0/1739411987 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f48010380 con 0x7f9f58101820 2024-04-22T21:00:43.934 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.218+0000 7f9f5d6c0700 1 -- 172.21.15.38:0/1739411987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f58101820 msgr2=0x7f9f58101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:43.934 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.218+0000 7f9f5d6c0700 1 --2- 172.21.15.38:0/1739411987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f58101820 0x7f9f58101c30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9f480098e0 tx=0x7f9f48009250 comp rx=0 tx=0).stop 2024-04-22T21:00:43.934 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.218+0000 7f9f5d6c0700 1 -- 172.21.15.38:0/1739411987 shutdown_connections 2024-04-22T21:00:43.934 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.218+0000 7f9f5d6c0700 1 --2- 172.21.15.38:0/1739411987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f58101820 0x7f9f58101c30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:43.934 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.218+0000 7f9f5d6c0700 1 -- 172.21.15.38:0/1739411987 >> 172.21.15.38:0/1739411987 conn(0x7f9f580fcf80 msgr2=0x7f9f580ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:43.934 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.218+0000 7f9f5d6c0700 1 -- 172.21.15.38:0/1739411987 shutdown_connections 2024-04-22T21:00:43.934 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.218+0000 7f9f5d6c0700 1 -- 172.21.15.38:0/1739411987 wait complete. 2024-04-22T21:00:43.934 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.219+0000 7f9f5d6c0700 1 Processor -- start 2024-04-22T21:00:43.934 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.219+0000 7f9f5d6c0700 1 -- start start 2024-04-22T21:00:43.934 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.219+0000 7f9f5d6c0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f58101820 0x7f9f580712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:43.934 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.219+0000 7f9f5d6c0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f58114410 con 0x7f9f58101820 2024-04-22T21:00:43.934 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.220+0000 7f9f56ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f58101820 0x7f9f580712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:43.935 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.220+0000 7f9f56ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f58101820 0x7f9f580712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49988/0 (socket says 172.21.15.38:49988) 2024-04-22T21:00:43.935 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.220+0000 7f9f56ffd700 1 -- 172.21.15.38:0/2863467626 learned_addr learned my addr 172.21.15.38:0/2863467626 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:43.935 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.220+0000 7f9f56ffd700 1 -- 172.21.15.38:0/2863467626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f480094f0 con 0x7f9f58101820 2024-04-22T21:00:43.935 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.220+0000 7f9f56ffd700 1 --2- 172.21.15.38:0/2863467626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f58101820 0x7f9f580712c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9f4800ba90 tx=0x7f9f48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:43.935 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.220+0000 7f9f3f7fe700 1 -- 172.21.15.38:0/2863467626 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f480107c0 con 0x7f9f58101820 2024-04-22T21:00:43.935 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.220+0000 7f9f3f7fe700 1 -- 172.21.15.38:0/2863467626 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9f48010db0 con 0x7f9f58101820 2024-04-22T21:00:43.935 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.220+0000 7f9f5d6c0700 1 -- 172.21.15.38:0/2863467626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f58114550 con 0x7f9f58101820 2024-04-22T21:00:43.935 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.220+0000 7f9f5d6c0700 1 -- 172.21.15.38:0/2863467626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f581149e0 con 0x7f9f58101820 2024-04-22T21:00:43.935 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.220+0000 7f9f3f7fe700 1 -- 172.21.15.38:0/2863467626 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f4801add0 con 0x7f9f58101820 2024-04-22T21:00:43.935 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.221+0000 7f9f3f7fe700 1 -- 172.21.15.38:0/2863467626 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f9f4802e440 con 0x7f9f58101820 2024-04-22T21:00:43.935 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.221+0000 7f9f3f7fe700 1 -- 172.21.15.38:0/2863467626 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9f48024740 con 0x7f9f58101820 2024-04-22T21:00:43.935 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.221+0000 7f9f5d6c0700 1 -- 172.21.15.38:0/2863467626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5804bed0 con 0x7f9f58101820 2024-04-22T21:00:43.935 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.223+0000 7f9f3f7fe700 1 -- 172.21.15.38:0/2863467626 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f9f48029070 con 0x7f9f58101820 2024-04-22T21:00:43.935 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.297+0000 7f9f5d6c0700 1 -- 172.21.15.38:0/2863467626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9f58103550 con 0x7f9f58101820 2024-04-22T21:00:43.935 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.297+0000 7f9f3f7fe700 1 -- 172.21.15.38:0/2863467626 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f9f4801a430 con 0x7f9f58101820 2024-04-22T21:00:43.936 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.299+0000 7f9f5d6c0700 1 -- 172.21.15.38:0/2863467626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f58101820 msgr2=0x7f9f580712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:43.936 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.299+0000 7f9f5d6c0700 1 --2- 172.21.15.38:0/2863467626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f58101820 0x7f9f580712c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9f4800ba90 tx=0x7f9f48009250 comp rx=0 tx=0).stop 2024-04-22T21:00:43.936 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.299+0000 7f9f5d6c0700 1 -- 172.21.15.38:0/2863467626 shutdown_connections 2024-04-22T21:00:43.936 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.299+0000 7f9f5d6c0700 1 --2- 172.21.15.38:0/2863467626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f58101820 0x7f9f580712c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:43.936 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.299+0000 7f9f5d6c0700 1 -- 172.21.15.38:0/2863467626 >> 172.21.15.38:0/2863467626 conn(0x7f9f580fcf80 msgr2=0x7f9f580fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:43.936 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.299+0000 7f9f5d6c0700 1 -- 172.21.15.38:0/2863467626 shutdown_connections 2024-04-22T21:00:43.936 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:43.299+0000 7f9f5d6c0700 1 -- 172.21.15.38:0/2863467626 wait complete. 2024-04-22T21:00:43.936 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (2/15)... 2024-04-22T21:00:46.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:46 smithi038 ceph-mon[26374]: Activating manager daemon smithi038.lxfgut 2024-04-22T21:00:46.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:46 smithi038 ceph-mon[26374]: mgrmap e2: smithi038.lxfgut(active, starting, since 0.00294378s) 2024-04-22T21:00:46.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:46 smithi038 ceph-mon[26374]: from='mgr.14100 172.21.15.38:0/3138578331' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T21:00:46.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:46 smithi038 ceph-mon[26374]: from='mgr.14100 172.21.15.38:0/3138578331' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T21:00:46.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:46 smithi038 ceph-mon[26374]: from='mgr.14100 172.21.15.38:0/3138578331' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T21:00:46.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:46 smithi038 ceph-mon[26374]: from='mgr.14100 172.21.15.38:0/3138578331' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2024-04-22T21:00:46.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:46 smithi038 ceph-mon[26374]: from='mgr.14100 172.21.15.38:0/3138578331' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr metadata", "who": "smithi038.lxfgut", "id": "smithi038.lxfgut"}]: dispatch 2024-04-22T21:00:46.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:46 smithi038 ceph-mon[26374]: Manager daemon smithi038.lxfgut is now available 2024-04-22T21:00:46.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:46 smithi038 ceph-mon[26374]: from='mgr.14100 172.21.15.38:0/3138578331' entity='mgr.smithi038.lxfgut' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.lxfgut/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:00:46.626 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:46 smithi038 ceph-mon[26374]: from='mgr.14100 172.21.15.38:0/3138578331' entity='mgr.smithi038.lxfgut' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.lxfgut/trash_purge_schedule"}]: dispatch 2024-04-22T21:00:46.626 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:46 smithi038 ceph-mon[26374]: from='mgr.14100 172.21.15.38:0/3138578331' entity='mgr.smithi038.lxfgut' 2024-04-22T21:00:46.626 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:46 smithi038 ceph-mon[26374]: from='mgr.14100 172.21.15.38:0/3138578331' entity='mgr.smithi038.lxfgut' 2024-04-22T21:00:46.626 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:46 smithi038 ceph-mon[26374]: from='mgr.14100 172.21.15.38:0/3138578331' entity='mgr.smithi038.lxfgut' 2024-04-22T21:00:46.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T21:00:46.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-22T21:00:46.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "3c5f5352-00eb-11ef-bc93-c7b262605968", 2024-04-22T21:00:46.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T21:00:46.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T21:00:46.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T21:00:46.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T21:00:46.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:46.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T21:00:46.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T21:00:46.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-04-22T21:00:46.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T21:00:46.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T21:00:46.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "smithi038" 2024-04-22T21:00:46.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T21:00:46.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-22T21:00:46.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T21:00:46.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T21:00:46.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-22T21:00:46.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T21:00:46.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:46.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T21:00:46.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T21:00:46.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T21:00:46.980 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T21:00:46.980 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T21:00:46.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T21:00:46.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T21:00:46.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T21:00:46.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:46.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T21:00:46.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T21:00:46.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T21:00:46.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T21:00:46.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T21:00:46.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T21:00:46.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T21:00:46.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T21:00:46.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T21:00:46.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:46.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T21:00:46.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T21:00:46.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T21:00:46.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T21:00:46.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:46.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T21:00:46.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T21:00:46.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T21:00:46.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T21:00:46.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T21:00:46.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T21:00:46.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T21:00:46.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T21:00:46.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T21:00:46.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:46.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T21:00:46.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T21:00:46.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T21:00:25.280583+0000", 2024-04-22T21:00:46.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T21:00:46.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:46.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T21:00:46.990 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-22T21:00:46.990 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.330+0000 7f8090135700 1 Processor -- start 2024-04-22T21:00:46.990 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.330+0000 7f8090135700 1 -- start start 2024-04-22T21:00:46.990 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.330+0000 7f8090135700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8088101dc0 0x7f80881021d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:46.990 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.330+0000 7f8090135700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8088102710 con 0x7f8088101dc0 2024-04-22T21:00:46.990 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.331+0000 7f808ded1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8088101dc0 0x7f80881021d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:46.990 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.331+0000 7f808ded1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8088101dc0 0x7f80881021d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50042/0 (socket says 172.21.15.38:50042) 2024-04-22T21:00:46.990 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.331+0000 7f808ded1700 1 -- 172.21.15.38:0/549892518 learned_addr learned my addr 172.21.15.38:0/549892518 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:46.990 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.331+0000 7f808ded1700 1 -- 172.21.15.38:0/549892518 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8088102850 con 0x7f8088101dc0 2024-04-22T21:00:46.990 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.331+0000 7f808ded1700 1 --2- 172.21.15.38:0/549892518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8088101dc0 0x7f80881021d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f808400bb80 tx=0x7f808400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15696ecf2465f23 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:46.990 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.331+0000 7f808cecf700 1 -- 172.21.15.38:0/549892518 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f808400dd00 con 0x7f8088101dc0 2024-04-22T21:00:46.990 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.331+0000 7f808cecf700 1 -- 172.21.15.38:0/549892518 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f808400de60 con 0x7f8088101dc0 2024-04-22T21:00:46.990 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.331+0000 7f808cecf700 1 -- 172.21.15.38:0/549892518 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80840188a0 con 0x7f8088101dc0 2024-04-22T21:00:46.991 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.332+0000 7f8090135700 1 -- 172.21.15.38:0/549892518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8088101dc0 msgr2=0x7f80881021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:46.991 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.332+0000 7f8090135700 1 --2- 172.21.15.38:0/549892518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8088101dc0 0x7f80881021d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f808400bb80 tx=0x7f808400b4f0 comp rx=0 tx=0).stop 2024-04-22T21:00:46.991 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.332+0000 7f8090135700 1 -- 172.21.15.38:0/549892518 shutdown_connections 2024-04-22T21:00:46.991 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.332+0000 7f8090135700 1 --2- 172.21.15.38:0/549892518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8088101dc0 0x7f80881021d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:46.991 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.332+0000 7f8090135700 1 -- 172.21.15.38:0/549892518 >> 172.21.15.38:0/549892518 conn(0x7f80880fc770 msgr2=0x7f80880feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:46.991 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.332+0000 7f8090135700 1 -- 172.21.15.38:0/549892518 shutdown_connections 2024-04-22T21:00:46.991 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.332+0000 7f8090135700 1 -- 172.21.15.38:0/549892518 wait complete. 2024-04-22T21:00:46.991 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.332+0000 7f8090135700 1 Processor -- start 2024-04-22T21:00:46.991 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.332+0000 7f8090135700 1 -- start start 2024-04-22T21:00:46.991 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.333+0000 7f8090135700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8088101dc0 0x7f8088113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:46.991 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.333+0000 7f8090135700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8088113e60 con 0x7f8088101dc0 2024-04-22T21:00:46.991 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.333+0000 7f808ded1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8088101dc0 0x7f8088113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:46.991 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.333+0000 7f808ded1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8088101dc0 0x7f8088113920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50050/0 (socket says 172.21.15.38:50050) 2024-04-22T21:00:46.991 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.333+0000 7f808ded1700 1 -- 172.21.15.38:0/3203783852 learned_addr learned my addr 172.21.15.38:0/3203783852 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:46.992 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.333+0000 7f808ded1700 1 -- 172.21.15.38:0/3203783852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808400b790 con 0x7f8088101dc0 2024-04-22T21:00:46.992 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.333+0000 7f808ded1700 1 --2- 172.21.15.38:0/3203783852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8088101dc0 0x7f8088113920 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8084000c00 tx=0x7f808400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:46.992 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.333+0000 7f807e7fc700 1 -- 172.21.15.38:0/3203783852 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8084010c00 con 0x7f8088101dc0 2024-04-22T21:00:46.992 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.333+0000 7f807e7fc700 1 -- 172.21.15.38:0/3203783852 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8084010d60 con 0x7f8088101dc0 2024-04-22T21:00:46.992 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.333+0000 7f8090135700 1 -- 172.21.15.38:0/3203783852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8088114060 con 0x7f8088101dc0 2024-04-22T21:00:46.992 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.333+0000 7f807e7fc700 1 -- 172.21.15.38:0/3203783852 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8084022850 con 0x7f8088101dc0 2024-04-22T21:00:46.992 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.333+0000 7f8090135700 1 -- 172.21.15.38:0/3203783852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8088114bd0 con 0x7f8088101dc0 2024-04-22T21:00:46.992 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.334+0000 7f807e7fc700 1 -- 172.21.15.38:0/3203783852 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 2) v1 ==== 43132+0+0 (secure 0 0 0) 0x7f8084022a50 con 0x7f8088101dc0 2024-04-22T21:00:46.992 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.334+0000 7f8090135700 1 -- 172.21.15.38:0/3203783852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f808805f5f0 con 0x7f8088101dc0 2024-04-22T21:00:46.992 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.334+0000 7f807e7fc700 1 -- 172.21.15.38:0/3203783852 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8084014070 con 0x7f8088101dc0 2024-04-22T21:00:46.992 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.336+0000 7f807e7fc700 1 -- 172.21.15.38:0/3203783852 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f808400f5b0 con 0x7f8088101dc0 2024-04-22T21:00:46.992 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.409+0000 7f8090135700 1 -- 172.21.15.38:0/3203783852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f808804bed0 con 0x7f8088101dc0 2024-04-22T21:00:46.992 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.409+0000 7f807e7fc700 1 -- 172.21.15.38:0/3203783852 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f808400f3d0 con 0x7f8088101dc0 2024-04-22T21:00:46.992 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.410+0000 7f8090135700 1 -- 172.21.15.38:0/3203783852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8088101dc0 msgr2=0x7f8088113920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:46.993 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.410+0000 7f8090135700 1 --2- 172.21.15.38:0/3203783852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8088101dc0 0x7f8088113920 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8084000c00 tx=0x7f808400b4f0 comp rx=0 tx=0).stop 2024-04-22T21:00:46.993 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.410+0000 7f8090135700 1 -- 172.21.15.38:0/3203783852 shutdown_connections 2024-04-22T21:00:46.993 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.410+0000 7f8090135700 1 --2- 172.21.15.38:0/3203783852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8088101dc0 0x7f8088113920 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:46.993 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.410+0000 7f8090135700 1 -- 172.21.15.38:0/3203783852 >> 172.21.15.38:0/3203783852 conn(0x7f80880fc770 msgr2=0x7f8088105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:46.993 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.410+0000 7f8090135700 1 -- 172.21.15.38:0/3203783852 shutdown_connections 2024-04-22T21:00:46.993 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:46.410+0000 7f8090135700 1 -- 172.21.15.38:0/3203783852 wait complete. 2024-04-22T21:00:46.993 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (3/15)... 2024-04-22T21:00:47.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:47 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3203783852' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T21:00:47.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:47 smithi038 ceph-mon[26374]: mgrmap e3: smithi038.lxfgut(active, since 1.00724s) 2024-04-22T21:00:49.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:49 smithi038 ceph-mon[26374]: mgrmap e4: smithi038.lxfgut(active, since 2s) 2024-04-22T21:00:50.235 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T21:00:50.235 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-22T21:00:50.235 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "3c5f5352-00eb-11ef-bc93-c7b262605968", 2024-04-22T21:00:50.235 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T21:00:50.235 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T21:00:50.235 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T21:00:50.235 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T21:00:50.235 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:50.235 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T21:00:50.235 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T21:00:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-04-22T21:00:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T21:00:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T21:00:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "smithi038" 2024-04-22T21:00:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T21:00:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-04-22T21:00:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T21:00:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T21:00:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-22T21:00:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T21:00:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T21:00:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T21:00:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T21:00:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T21:00:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T21:00:50.244 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T21:00:50.244 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T21:00:50.244 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T21:00:50.244 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:50.244 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T21:00:50.244 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T21:00:50.245 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T21:00:50.245 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T21:00:50.245 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T21:00:50.245 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T21:00:50.245 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T21:00:50.245 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T21:00:50.245 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T21:00:50.245 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:50.245 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T21:00:50.245 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T21:00:50.245 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T21:00:50.245 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T21:00:50.245 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:50.245 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T21:00:50.246 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T21:00:50.246 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T21:00:50.246 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T21:00:50.246 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T21:00:50.246 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T21:00:50.246 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T21:00:50.246 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T21:00:50.246 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T21:00:50.246 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:50.246 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T21:00:50.246 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T21:00:50.246 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T21:00:25.280583+0000", 2024-04-22T21:00:50.246 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T21:00:50.246 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T21:00:50.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T21:00:50.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-22T21:00:50.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.380+0000 7f448db95700 1 Processor -- start 2024-04-22T21:00:50.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.380+0000 7f448db95700 1 -- start start 2024-04-22T21:00:50.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.380+0000 7f448db95700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4488076ac0 0x7f4488105600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:50.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.380+0000 7f448db95700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4488076ed0 con 0x7f4488076ac0 2024-04-22T21:00:50.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.380+0000 7f44877fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4488076ac0 0x7f4488105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:50.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.380+0000 7f44877fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4488076ac0 0x7f4488105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50056/0 (socket says 172.21.15.38:50056) 2024-04-22T21:00:50.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.380+0000 7f44877fe700 1 -- 172.21.15.38:0/1479179255 learned_addr learned my addr 172.21.15.38:0/1479179255 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:50.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.381+0000 7f44877fe700 1 -- 172.21.15.38:0/1479179255 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4488105b40 con 0x7f4488076ac0 2024-04-22T21:00:50.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.381+0000 7f44877fe700 1 --2- 172.21.15.38:0/1479179255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4488076ac0 0x7f4488105600 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f44780098e0 tx=0x7f4478009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e05be1de5f279c71 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:50.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.381+0000 7f44867fc700 1 -- 172.21.15.38:0/1479179255 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f447800b8f0 con 0x7f4488076ac0 2024-04-22T21:00:50.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.381+0000 7f44867fc700 1 -- 172.21.15.38:0/1479179255 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4478011420 con 0x7f4488076ac0 2024-04-22T21:00:50.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.381+0000 7f44867fc700 1 -- 172.21.15.38:0/1479179255 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4478010540 con 0x7f4488076ac0 2024-04-22T21:00:50.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.381+0000 7f448db95700 1 -- 172.21.15.38:0/1479179255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4488076ac0 msgr2=0x7f4488105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:50.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.382+0000 7f448db95700 1 --2- 172.21.15.38:0/1479179255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4488076ac0 0x7f4488105600 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f44780098e0 tx=0x7f4478009250 comp rx=0 tx=0).stop 2024-04-22T21:00:50.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.382+0000 7f448db95700 1 -- 172.21.15.38:0/1479179255 shutdown_connections 2024-04-22T21:00:50.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.382+0000 7f448db95700 1 --2- 172.21.15.38:0/1479179255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4488076ac0 0x7f4488105600 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:50.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.382+0000 7f448db95700 1 -- 172.21.15.38:0/1479179255 >> 172.21.15.38:0/1479179255 conn(0x7f44880fd050 msgr2=0x7f44880ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:50.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.382+0000 7f448db95700 1 -- 172.21.15.38:0/1479179255 shutdown_connections 2024-04-22T21:00:50.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.382+0000 7f448db95700 1 -- 172.21.15.38:0/1479179255 wait complete. 2024-04-22T21:00:50.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.382+0000 7f448db95700 1 Processor -- start 2024-04-22T21:00:50.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.382+0000 7f448db95700 1 -- start start 2024-04-22T21:00:50.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.383+0000 7f448db95700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4488076ac0 0x7f4488102f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:50.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.383+0000 7f448db95700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4488103490 con 0x7f4488076ac0 2024-04-22T21:00:50.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.383+0000 7f44877fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4488076ac0 0x7f4488102f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:50.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.383+0000 7f44877fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4488076ac0 0x7f4488102f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50058/0 (socket says 172.21.15.38:50058) 2024-04-22T21:00:50.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.383+0000 7f44877fe700 1 -- 172.21.15.38:0/1676019940 learned_addr learned my addr 172.21.15.38:0/1676019940 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:50.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.383+0000 7f44877fe700 1 -- 172.21.15.38:0/1676019940 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44780094f0 con 0x7f4488076ac0 2024-04-22T21:00:50.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.383+0000 7f44877fe700 1 --2- 172.21.15.38:0/1676019940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4488076ac0 0x7f4488102f50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f44780094c0 tx=0x7f4478009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:50.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.383+0000 7f446ffff700 1 -- 172.21.15.38:0/1676019940 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4478010800 con 0x7f4488076ac0 2024-04-22T21:00:50.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.383+0000 7f446ffff700 1 -- 172.21.15.38:0/1676019940 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4478010df0 con 0x7f4488076ac0 2024-04-22T21:00:50.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.383+0000 7f448db95700 1 -- 172.21.15.38:0/1676019940 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4488101560 con 0x7f4488076ac0 2024-04-22T21:00:50.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.384+0000 7f446ffff700 1 -- 172.21.15.38:0/1676019940 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4478024e50 con 0x7f4488076ac0 2024-04-22T21:00:50.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.384+0000 7f448db95700 1 -- 172.21.15.38:0/1676019940 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4488101a50 con 0x7f4488076ac0 2024-04-22T21:00:50.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.384+0000 7f446ffff700 1 -- 172.21.15.38:0/1676019940 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) v1 ==== 43365+0+0 (secure 0 0 0) 0x7f447801a470 con 0x7f4488076ac0 2024-04-22T21:00:50.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.385+0000 7f446ffff700 1 --2- 172.21.15.38:0/1676019940 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f4470036740 0x7f4470038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:50.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.385+0000 7f446ffff700 1 -- 172.21.15.38:0/1676019940 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f447804a510 con 0x7f4488076ac0 2024-04-22T21:00:50.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.385+0000 7f448db95700 1 -- 172.21.15.38:0/1676019940 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f448804bed0 con 0x7f4488076ac0 2024-04-22T21:00:50.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.385+0000 7f4486ffd700 1 --2- 172.21.15.38:0/1676019940 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f4470036740 0x7f4470038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:50.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.385+0000 7f4486ffd700 1 --2- 172.21.15.38:0/1676019940 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f4470036740 0x7f4470038bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f447c009990 tx=0x7f447c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:50.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.390+0000 7f446ffff700 1 -- 172.21.15.38:0/1676019940 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f447801a720 con 0x7f4488076ac0 2024-04-22T21:00:50.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.663+0000 7f448db95700 1 -- 172.21.15.38:0/1676019940 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4488102050 con 0x7f4488076ac0 2024-04-22T21:00:50.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.664+0000 7f446ffff700 1 -- 172.21.15.38:0/1676019940 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f4478048030 con 0x7f4488076ac0 2024-04-22T21:00:50.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.666+0000 7f448db95700 1 -- 172.21.15.38:0/1676019940 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f4470036740 msgr2=0x7f4470038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:50.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.666+0000 7f448db95700 1 --2- 172.21.15.38:0/1676019940 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f4470036740 0x7f4470038bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f447c009990 tx=0x7f447c006e40 comp rx=0 tx=0).stop 2024-04-22T21:00:50.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.666+0000 7f448db95700 1 -- 172.21.15.38:0/1676019940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4488076ac0 msgr2=0x7f4488102f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:50.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.666+0000 7f448db95700 1 --2- 172.21.15.38:0/1676019940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4488076ac0 0x7f4488102f50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f44780094c0 tx=0x7f4478009250 comp rx=0 tx=0).stop 2024-04-22T21:00:50.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.666+0000 7f448db95700 1 -- 172.21.15.38:0/1676019940 shutdown_connections 2024-04-22T21:00:50.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.666+0000 7f448db95700 1 --2- 172.21.15.38:0/1676019940 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f4470036740 0x7f4470038bf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:50.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.666+0000 7f448db95700 1 --2- 172.21.15.38:0/1676019940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4488076ac0 0x7f4488102f50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:50.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.666+0000 7f448db95700 1 -- 172.21.15.38:0/1676019940 >> 172.21.15.38:0/1676019940 conn(0x7f44880fd050 msgr2=0x7f44880fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:50.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.666+0000 7f448db95700 1 -- 172.21.15.38:0/1676019940 shutdown_connections 2024-04-22T21:00:50.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:49.666+0000 7f448db95700 1 -- 172.21.15.38:0/1676019940 wait complete. 2024-04-22T21:00:50.250 INFO:teuthology.orchestra.run.smithi038.stdout:mgr is available 2024-04-22T21:00:50.574 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:50 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1676019940' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T21:00:51.470 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:51 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3787783953' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T21:00:51.479 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T21:00:51.479 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T21:00:51.479 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout fsid = 3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T21:00:51.479 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T21:00:51.479 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T21:00:51.479 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T21:00:51.479 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T21:00:51.479 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T21:00:51.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T21:00:51.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T21:00:51.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T21:00:51.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T21:00:51.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T21:00:51.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T21:00:51.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T21:00:51.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T21:00:51.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T21:00:51.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.684+0000 7ff0d04fd700 1 Processor -- start 2024-04-22T21:00:51.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.684+0000 7ff0d04fd700 1 -- start start 2024-04-22T21:00:51.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.684+0000 7ff0d04fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c8101820 0x7ff0c8101c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:51.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.684+0000 7ff0d04fd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c8102170 con 0x7ff0c8101820 2024-04-22T21:00:51.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.684+0000 7ff0ce299700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c8101820 0x7ff0c8101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:51.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.684+0000 7ff0ce299700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c8101820 0x7ff0c8101c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50062/0 (socket says 172.21.15.38:50062) 2024-04-22T21:00:51.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.684+0000 7ff0ce299700 1 -- 172.21.15.38:0/224908983 learned_addr learned my addr 172.21.15.38:0/224908983 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:51.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.685+0000 7ff0ce299700 1 -- 172.21.15.38:0/224908983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c81022b0 con 0x7ff0c8101820 2024-04-22T21:00:51.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.685+0000 7ff0ce299700 1 --2- 172.21.15.38:0/224908983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c8101820 0x7ff0c8101c30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff0c40098e0 tx=0x7ff0c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b698d44efcf8fe7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:51.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.685+0000 7ff0cd297700 1 -- 172.21.15.38:0/224908983 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0c400b860 con 0x7ff0c8101820 2024-04-22T21:00:51.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.685+0000 7ff0cd297700 1 -- 172.21.15.38:0/224908983 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff0c400be50 con 0x7ff0c8101820 2024-04-22T21:00:51.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.685+0000 7ff0cd297700 1 -- 172.21.15.38:0/224908983 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0c4010380 con 0x7ff0c8101820 2024-04-22T21:00:51.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.686+0000 7ff0d04fd700 1 -- 172.21.15.38:0/224908983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c8101820 msgr2=0x7ff0c8101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:51.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.686+0000 7ff0d04fd700 1 --2- 172.21.15.38:0/224908983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c8101820 0x7ff0c8101c30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff0c40098e0 tx=0x7ff0c4009250 comp rx=0 tx=0).stop 2024-04-22T21:00:51.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.686+0000 7ff0d04fd700 1 -- 172.21.15.38:0/224908983 shutdown_connections 2024-04-22T21:00:51.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.686+0000 7ff0d04fd700 1 --2- 172.21.15.38:0/224908983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c8101820 0x7ff0c8101c30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:51.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.686+0000 7ff0d04fd700 1 -- 172.21.15.38:0/224908983 >> 172.21.15.38:0/224908983 conn(0x7ff0c80fcfe0 msgr2=0x7ff0c80ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:51.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.686+0000 7ff0d04fd700 1 -- 172.21.15.38:0/224908983 shutdown_connections 2024-04-22T21:00:51.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.686+0000 7ff0d04fd700 1 -- 172.21.15.38:0/224908983 wait complete. 2024-04-22T21:00:51.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.686+0000 7ff0d04fd700 1 Processor -- start 2024-04-22T21:00:51.482 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.686+0000 7ff0d04fd700 1 -- start start 2024-04-22T21:00:51.482 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.686+0000 7ff0d04fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c8101820 0x7ff0c8076b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:51.482 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.686+0000 7ff0d04fd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c81143c0 con 0x7ff0c8101820 2024-04-22T21:00:51.482 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.687+0000 7ff0ce299700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c8101820 0x7ff0c8076b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:51.482 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.687+0000 7ff0ce299700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c8101820 0x7ff0c8076b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50068/0 (socket says 172.21.15.38:50068) 2024-04-22T21:00:51.482 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.687+0000 7ff0ce299700 1 -- 172.21.15.38:0/3787783953 learned_addr learned my addr 172.21.15.38:0/3787783953 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:51.482 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.687+0000 7ff0ce299700 1 -- 172.21.15.38:0/3787783953 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c40094f0 con 0x7ff0c8101820 2024-04-22T21:00:51.482 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.687+0000 7ff0ce299700 1 --2- 172.21.15.38:0/3787783953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c8101820 0x7ff0c8076b90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4006b00 tx=0x7ff0c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:51.482 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.687+0000 7ff0beffd700 1 -- 172.21.15.38:0/3787783953 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0c4010760 con 0x7ff0c8101820 2024-04-22T21:00:51.482 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.687+0000 7ff0d04fd700 1 -- 172.21.15.38:0/3787783953 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c8114500 con 0x7ff0c8101820 2024-04-22T21:00:51.482 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.687+0000 7ff0beffd700 1 -- 172.21.15.38:0/3787783953 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff0c4010d50 con 0x7ff0c8101820 2024-04-22T21:00:51.482 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.687+0000 7ff0beffd700 1 -- 172.21.15.38:0/3787783953 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0c401ad40 con 0x7ff0c8101820 2024-04-22T21:00:51.482 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.687+0000 7ff0d04fd700 1 -- 172.21.15.38:0/3787783953 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c81149f0 con 0x7ff0c8101820 2024-04-22T21:00:51.483 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.688+0000 7ff0beffd700 1 -- 172.21.15.38:0/3787783953 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) v1 ==== 43365+0+0 (secure 0 0 0) 0x7ff0c402e440 con 0x7ff0c8101820 2024-04-22T21:00:51.483 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.688+0000 7ff0d04fd700 1 -- 172.21.15.38:0/3787783953 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c804bed0 con 0x7ff0c8101820 2024-04-22T21:00:51.483 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.688+0000 7ff0beffd700 1 --2- 172.21.15.38:0/3787783953 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7ff0b4036740 0x7ff0b4038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:51.483 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.688+0000 7ff0beffd700 1 -- 172.21.15.38:0/3787783953 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff0c404a680 con 0x7ff0c8101820 2024-04-22T21:00:51.483 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.688+0000 7ff0cda98700 1 --2- 172.21.15.38:0/3787783953 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7ff0b4036740 0x7ff0b4038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:51.483 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.689+0000 7ff0cda98700 1 --2- 172.21.15.38:0/3787783953 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7ff0b4036740 0x7ff0b4038bf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8009990 tx=0x7ff0b8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:51.483 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.693+0000 7ff0beffd700 1 -- 172.21.15.38:0/3787783953 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff0c4018350 con 0x7ff0c8101820 2024-04-22T21:00:51.483 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.888+0000 7ff0d04fd700 1 -- 172.21.15.38:0/3787783953 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7ff0c81035a0 con 0x7ff0c8101820 2024-04-22T21:00:51.483 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.889+0000 7ff0beffd700 1 -- 172.21.15.38:0/3787783953 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7ff0c4018350 con 0x7ff0c8101820 2024-04-22T21:00:51.483 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.891+0000 7ff0d04fd700 1 -- 172.21.15.38:0/3787783953 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7ff0b4036740 msgr2=0x7ff0b4038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:51.483 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.891+0000 7ff0d04fd700 1 --2- 172.21.15.38:0/3787783953 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7ff0b4036740 0x7ff0b4038bf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8009990 tx=0x7ff0b8006e40 comp rx=0 tx=0).stop 2024-04-22T21:00:51.483 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.891+0000 7ff0d04fd700 1 -- 172.21.15.38:0/3787783953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c8101820 msgr2=0x7ff0c8076b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:51.483 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.891+0000 7ff0d04fd700 1 --2- 172.21.15.38:0/3787783953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c8101820 0x7ff0c8076b90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4006b00 tx=0x7ff0c4009250 comp rx=0 tx=0).stop 2024-04-22T21:00:51.483 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.891+0000 7ff0d04fd700 1 -- 172.21.15.38:0/3787783953 shutdown_connections 2024-04-22T21:00:51.484 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.891+0000 7ff0d04fd700 1 --2- 172.21.15.38:0/3787783953 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7ff0b4036740 0x7ff0b4038bf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:51.484 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.891+0000 7ff0d04fd700 1 --2- 172.21.15.38:0/3787783953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c8101820 0x7ff0c8076b90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:51.484 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.891+0000 7ff0d04fd700 1 -- 172.21.15.38:0/3787783953 >> 172.21.15.38:0/3787783953 conn(0x7ff0c80fcfe0 msgr2=0x7ff0c80fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:51.484 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.891+0000 7ff0d04fd700 1 -- 172.21.15.38:0/3787783953 shutdown_connections 2024-04-22T21:00:51.484 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:50.891+0000 7ff0d04fd700 1 -- 172.21.15.38:0/3787783953 wait complete. 2024-04-22T21:00:51.484 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling cephadm module... 2024-04-22T21:00:52.473 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:52 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/247541615' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-22T21:00:52.777 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.951+0000 7f01cecf7700 1 Processor -- start 2024-04-22T21:00:52.777 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.951+0000 7f01cecf7700 1 -- start start 2024-04-22T21:00:52.777 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.952+0000 7f01cecf7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01c8076b40 0x7f01c8105550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:52.777 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.952+0000 7f01cecf7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c8076f50 con 0x7f01c8076b40 2024-04-22T21:00:52.777 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.952+0000 7f01cca93700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01c8076b40 0x7f01c8105550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:52.777 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.952+0000 7f01cca93700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01c8076b40 0x7f01c8105550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50084/0 (socket says 172.21.15.38:50084) 2024-04-22T21:00:52.777 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.952+0000 7f01cca93700 1 -- 172.21.15.38:0/3292185808 learned_addr learned my addr 172.21.15.38:0/3292185808 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:52.777 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.952+0000 7f01cca93700 1 -- 172.21.15.38:0/3292185808 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c8105a90 con 0x7f01c8076b40 2024-04-22T21:00:52.777 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.953+0000 7f01cca93700 1 --2- 172.21.15.38:0/3292185808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01c8076b40 0x7f01c8105550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f01bc0098e0 tx=0x7f01bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9455139d69a3a7b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:52.777 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.953+0000 7f01c77fe700 1 -- 172.21.15.38:0/3292185808 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01bc00b8f0 con 0x7f01c8076b40 2024-04-22T21:00:52.778 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.953+0000 7f01c77fe700 1 -- 172.21.15.38:0/3292185808 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f01bc011420 con 0x7f01c8076b40 2024-04-22T21:00:52.778 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.953+0000 7f01c77fe700 1 -- 172.21.15.38:0/3292185808 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01bc010540 con 0x7f01c8076b40 2024-04-22T21:00:52.778 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.953+0000 7f01cecf7700 1 -- 172.21.15.38:0/3292185808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01c8076b40 msgr2=0x7f01c8105550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:52.778 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.953+0000 7f01cecf7700 1 --2- 172.21.15.38:0/3292185808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01c8076b40 0x7f01c8105550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f01bc0098e0 tx=0x7f01bc009250 comp rx=0 tx=0).stop 2024-04-22T21:00:52.778 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.953+0000 7f01cecf7700 1 -- 172.21.15.38:0/3292185808 shutdown_connections 2024-04-22T21:00:52.778 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.953+0000 7f01cecf7700 1 --2- 172.21.15.38:0/3292185808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01c8076b40 0x7f01c8105550 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:52.778 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.953+0000 7f01cecf7700 1 -- 172.21.15.38:0/3292185808 >> 172.21.15.38:0/3292185808 conn(0x7f01c80fd040 msgr2=0x7f01c80ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:52.778 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.953+0000 7f01cecf7700 1 -- 172.21.15.38:0/3292185808 shutdown_connections 2024-04-22T21:00:52.778 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.954+0000 7f01cecf7700 1 -- 172.21.15.38:0/3292185808 wait complete. 2024-04-22T21:00:52.778 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.954+0000 7f01cecf7700 1 Processor -- start 2024-04-22T21:00:52.778 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.954+0000 7f01cecf7700 1 -- start start 2024-04-22T21:00:52.779 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.954+0000 7f01cecf7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01c8076b40 0x7f01c8111740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:52.779 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.954+0000 7f01cecf7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c8111c80 con 0x7f01c8076b40 2024-04-22T21:00:52.779 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.955+0000 7f01cca93700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01c8076b40 0x7f01c8111740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:52.779 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.955+0000 7f01cca93700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01c8076b40 0x7f01c8111740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50094/0 (socket says 172.21.15.38:50094) 2024-04-22T21:00:52.779 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.955+0000 7f01cca93700 1 -- 172.21.15.38:0/247541615 learned_addr learned my addr 172.21.15.38:0/247541615 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:52.779 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.955+0000 7f01cca93700 1 -- 172.21.15.38:0/247541615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01bc0094f0 con 0x7f01c8076b40 2024-04-22T21:00:52.779 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.955+0000 7f01cca93700 1 --2- 172.21.15.38:0/247541615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01c8076b40 0x7f01c8111740 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f01bc000c00 tx=0x7f01bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:52.779 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.955+0000 7f01c57fa700 1 -- 172.21.15.38:0/247541615 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01bc0107b0 con 0x7f01c8076b40 2024-04-22T21:00:52.779 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.955+0000 7f01cecf7700 1 -- 172.21.15.38:0/247541615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c8111e80 con 0x7f01c8076b40 2024-04-22T21:00:52.779 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.955+0000 7f01c57fa700 1 -- 172.21.15.38:0/247541615 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f01bc010da0 con 0x7f01c8076b40 2024-04-22T21:00:52.780 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.955+0000 7f01c57fa700 1 -- 172.21.15.38:0/247541615 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01bc01add0 con 0x7f01c8076b40 2024-04-22T21:00:52.780 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.955+0000 7f01cecf7700 1 -- 172.21.15.38:0/247541615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c81129f0 con 0x7f01c8076b40 2024-04-22T21:00:52.780 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.956+0000 7f01c57fa700 1 -- 172.21.15.38:0/247541615 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) v1 ==== 43365+0+0 (secure 0 0 0) 0x7f01bc010910 con 0x7f01c8076b40 2024-04-22T21:00:52.780 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.956+0000 7f01c57fa700 1 --2- 172.21.15.38:0/247541615 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f01b8036740 0x7f01b8038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:52.780 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.956+0000 7f01cecf7700 1 -- 172.21.15.38:0/247541615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c805f5f0 con 0x7f01c8076b40 2024-04-22T21:00:52.780 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.957+0000 7f01c57fa700 1 -- 172.21.15.38:0/247541615 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f01bc04ac40 con 0x7f01c8076b40 2024-04-22T21:00:52.780 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.957+0000 7f01c7fff700 1 --2- 172.21.15.38:0/247541615 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f01b8036740 0x7f01b8038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:52.780 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.957+0000 7f01c7fff700 1 --2- 172.21.15.38:0/247541615 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f01b8036740 0x7f01b8038bf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009990 tx=0x7f01b4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:52.780 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:51.961+0000 7f01c57fa700 1 -- 172.21.15.38:0/247541615 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f01bc01a5c0 con 0x7f01c8076b40 2024-04-22T21:00:52.780 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:52.213+0000 7f01cecf7700 1 -- 172.21.15.38:0/247541615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f01c804bed0 con 0x7f01c8076b40 2024-04-22T21:00:52.780 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:52.314+0000 7f01c57fa700 1 -- 172.21.15.38:0/247541615 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 5) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f01bc017d90 con 0x7f01c8076b40 2024-04-22T21:00:52.780 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:52.314+0000 7f01c57fa700 1 -- 172.21.15.38:0/247541615 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f01bc02c550 con 0x7f01c8076b40 2024-04-22T21:00:52.781 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:52.316+0000 7f01cecf7700 1 -- 172.21.15.38:0/247541615 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f01b8036740 msgr2=0x7f01b8038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:52.781 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:52.316+0000 7f01cecf7700 1 --2- 172.21.15.38:0/247541615 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f01b8036740 0x7f01b8038bf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009990 tx=0x7f01b4006e40 comp rx=0 tx=0).stop 2024-04-22T21:00:52.781 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:52.316+0000 7f01cecf7700 1 -- 172.21.15.38:0/247541615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01c8076b40 msgr2=0x7f01c8111740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:52.781 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:52.316+0000 7f01cecf7700 1 --2- 172.21.15.38:0/247541615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01c8076b40 0x7f01c8111740 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f01bc000c00 tx=0x7f01bc009250 comp rx=0 tx=0).stop 2024-04-22T21:00:52.781 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:52.317+0000 7f01cecf7700 1 -- 172.21.15.38:0/247541615 shutdown_connections 2024-04-22T21:00:52.781 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:52.317+0000 7f01cecf7700 1 --2- 172.21.15.38:0/247541615 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f01b8036740 0x7f01b8038bf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:52.781 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:52.317+0000 7f01cecf7700 1 --2- 172.21.15.38:0/247541615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01c8076b40 0x7f01c8111740 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:52.781 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:52.317+0000 7f01cecf7700 1 -- 172.21.15.38:0/247541615 >> 172.21.15.38:0/247541615 conn(0x7f01c80fd040 msgr2=0x7f01c80fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:52.781 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:52.317+0000 7f01cecf7700 1 -- 172.21.15.38:0/247541615 shutdown_connections 2024-04-22T21:00:52.781 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:52.317+0000 7f01cecf7700 1 -- 172.21.15.38:0/247541615 wait complete. 2024-04-22T21:00:53.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:53 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/247541615' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-22T21:00:53.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:53 smithi038 ceph-mon[26374]: mgrmap e5: smithi038.lxfgut(active, since 6s) 2024-04-22T21:00:54.041 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-22T21:00:54.042 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-22T21:00:54.042 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T21:00:54.042 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "active_name": "smithi038.lxfgut", 2024-04-22T21:00:54.042 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T21:00:54.042 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-22T21:00:54.042 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.199+0000 7f25f828f700 1 Processor -- start 2024-04-22T21:00:54.042 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.199+0000 7f25f828f700 1 -- start start 2024-04-22T21:00:54.042 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.199+0000 7f25f828f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f0101820 0x7f25f0101c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:54.042 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.199+0000 7f25f828f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f0102170 con 0x7f25f0101820 2024-04-22T21:00:54.042 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.199+0000 7f25f602b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f0101820 0x7f25f0101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:54.042 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.199+0000 7f25f602b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f0101820 0x7f25f0101c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43000/0 (socket says 172.21.15.38:43000) 2024-04-22T21:00:54.042 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.199+0000 7f25f602b700 1 -- 172.21.15.38:0/3380018738 learned_addr learned my addr 172.21.15.38:0/3380018738 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:54.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.200+0000 7f25f602b700 1 -- 172.21.15.38:0/3380018738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f01022b0 con 0x7f25f0101820 2024-04-22T21:00:54.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.200+0000 7f25f602b700 1 --2- 172.21.15.38:0/3380018738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f0101820 0x7f25f0101c30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0098e0 tx=0x7f25ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd2162d0413a10f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:54.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.200+0000 7f25f5029700 1 -- 172.21.15.38:0/3380018738 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25ec00b860 con 0x7f25f0101820 2024-04-22T21:00:54.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.200+0000 7f25f5029700 1 -- 172.21.15.38:0/3380018738 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f25ec00be50 con 0x7f25f0101820 2024-04-22T21:00:54.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.200+0000 7f25f5029700 1 -- 172.21.15.38:0/3380018738 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25ec010380 con 0x7f25f0101820 2024-04-22T21:00:54.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.200+0000 7f25f828f700 1 -- 172.21.15.38:0/3380018738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f0101820 msgr2=0x7f25f0101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:54.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.200+0000 7f25f828f700 1 --2- 172.21.15.38:0/3380018738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f0101820 0x7f25f0101c30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0098e0 tx=0x7f25ec009250 comp rx=0 tx=0).stop 2024-04-22T21:00:54.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.201+0000 7f25f828f700 1 -- 172.21.15.38:0/3380018738 shutdown_connections 2024-04-22T21:00:54.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.201+0000 7f25f828f700 1 --2- 172.21.15.38:0/3380018738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f0101820 0x7f25f0101c30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:54.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.201+0000 7f25f828f700 1 -- 172.21.15.38:0/3380018738 >> 172.21.15.38:0/3380018738 conn(0x7f25f00fcf80 msgr2=0x7f25f00ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:54.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.201+0000 7f25f828f700 1 -- 172.21.15.38:0/3380018738 shutdown_connections 2024-04-22T21:00:54.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.201+0000 7f25f828f700 1 -- 172.21.15.38:0/3380018738 wait complete. 2024-04-22T21:00:54.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.201+0000 7f25f828f700 1 Processor -- start 2024-04-22T21:00:54.043 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.201+0000 7f25f828f700 1 -- start start 2024-04-22T21:00:54.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.201+0000 7f25f828f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f0101820 0x7f25f0113900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:54.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.201+0000 7f25f828f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f0113e40 con 0x7f25f0101820 2024-04-22T21:00:54.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.201+0000 7f25f602b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f0101820 0x7f25f0113900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:00:54.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.202+0000 7f25f602b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f0101820 0x7f25f0113900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43012/0 (socket says 172.21.15.38:43012) 2024-04-22T21:00:54.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.202+0000 7f25f602b700 1 -- 172.21.15.38:0/87035272 learned_addr learned my addr 172.21.15.38:0/87035272 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:00:54.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.202+0000 7f25f602b700 1 -- 172.21.15.38:0/87035272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec0094f0 con 0x7f25f0101820 2024-04-22T21:00:54.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.202+0000 7f25f602b700 1 --2- 172.21.15.38:0/87035272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f0101820 0x7f25f0113900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f25ec00ba90 tx=0x7f25ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:00:54.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.202+0000 7f25e6ffd700 1 -- 172.21.15.38:0/87035272 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25ec0107c0 con 0x7f25f0101820 2024-04-22T21:00:54.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.202+0000 7f25e6ffd700 1 -- 172.21.15.38:0/87035272 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f25ec010db0 con 0x7f25f0101820 2024-04-22T21:00:54.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.202+0000 7f25f828f700 1 -- 172.21.15.38:0/87035272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f0114040 con 0x7f25f0101820 2024-04-22T21:00:54.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.202+0000 7f25e6ffd700 1 -- 172.21.15.38:0/87035272 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25ec01add0 con 0x7f25f0101820 2024-04-22T21:00:54.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.202+0000 7f25f828f700 1 -- 172.21.15.38:0/87035272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f0114bb0 con 0x7f25f0101820 2024-04-22T21:00:54.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.203+0000 7f25e6ffd700 1 -- 172.21.15.38:0/87035272 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 5) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f25ec02e440 con 0x7f25f0101820 2024-04-22T21:00:54.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.203+0000 7f25f828f700 1 -- 172.21.15.38:0/87035272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f005f5f0 con 0x7f25f0101820 2024-04-22T21:00:54.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.203+0000 7f25e6ffd700 1 --2- 172.21.15.38:0/87035272 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f25dc0367d0 0x7f25dc038c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:00:54.044 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.203+0000 7f25f582a700 1 -- 172.21.15.38:0/87035272 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f25dc0367d0 msgr2=0x7f25dc038c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2787760150 2024-04-22T21:00:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.203+0000 7f25e6ffd700 1 -- 172.21.15.38:0/87035272 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f25ec04a600 con 0x7f25f0101820 2024-04-22T21:00:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.203+0000 7f25f582a700 1 --2- 172.21.15.38:0/87035272 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f25dc0367d0 0x7f25dc038c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T21:00:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.208+0000 7f25e6ffd700 1 -- 172.21.15.38:0/87035272 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f25ec018940 con 0x7f25f0101820 2024-04-22T21:00:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.404+0000 7f25f582a700 1 -- 172.21.15.38:0/87035272 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f25dc0367d0 msgr2=0x7f25dc038c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2787760150 2024-04-22T21:00:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.404+0000 7f25f582a700 1 --2- 172.21.15.38:0/87035272 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f25dc0367d0 0x7f25dc038c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T21:00:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.460+0000 7f25f828f700 1 -- 172.21.15.38:0/87035272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f25f01156d0 con 0x7f25f0101820 2024-04-22T21:00:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.460+0000 7f25e6ffd700 1 -- 172.21.15.38:0/87035272 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f25ec018350 con 0x7f25f0101820 2024-04-22T21:00:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.462+0000 7f25f828f700 1 -- 172.21.15.38:0/87035272 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f25dc0367d0 msgr2=0x7f25dc038c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T21:00:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.462+0000 7f25f828f700 1 --2- 172.21.15.38:0/87035272 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f25dc0367d0 0x7f25dc038c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.462+0000 7f25f828f700 1 -- 172.21.15.38:0/87035272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f0101820 msgr2=0x7f25f0113900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:00:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.462+0000 7f25f828f700 1 --2- 172.21.15.38:0/87035272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f0101820 0x7f25f0113900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f25ec00ba90 tx=0x7f25ec009250 comp rx=0 tx=0).stop 2024-04-22T21:00:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.462+0000 7f25f828f700 1 -- 172.21.15.38:0/87035272 shutdown_connections 2024-04-22T21:00:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.462+0000 7f25f828f700 1 --2- 172.21.15.38:0/87035272 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7f25dc0367d0 0x7f25dc038c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.462+0000 7f25f828f700 1 --2- 172.21.15.38:0/87035272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f0101820 0x7f25f0113900 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:00:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.462+0000 7f25f828f700 1 -- 172.21.15.38:0/87035272 >> 172.21.15.38:0/87035272 conn(0x7f25f00fcf80 msgr2=0x7f25f0105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:00:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.462+0000 7f25f828f700 1 -- 172.21.15.38:0/87035272 shutdown_connections 2024-04-22T21:00:54.046 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:53.463+0000 7f25f828f700 1 -- 172.21.15.38:0/87035272 wait complete. 2024-04-22T21:00:54.046 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for the mgr to restart... 2024-04-22T21:00:54.046 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr epoch 5... 2024-04-22T21:00:54.598 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:54 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/87035272' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T21:00:59.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:59 smithi038 ceph-mon[26374]: Active manager daemon smithi038.lxfgut restarted 2024-04-22T21:00:59.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:59 smithi038 ceph-mon[26374]: Activating manager daemon smithi038.lxfgut 2024-04-22T21:00:59.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:59 smithi038 ceph-mon[26374]: osdmap e2: 0 total, 0 up, 0 in 2024-04-22T21:00:59.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:59 smithi038 ceph-mon[26374]: mgrmap e6: smithi038.lxfgut(active, starting, since 0.0082413s) 2024-04-22T21:00:59.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:59 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2024-04-22T21:00:59.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:59 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr metadata", "who": "smithi038.lxfgut", "id": "smithi038.lxfgut"}]: dispatch 2024-04-22T21:00:59.126 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:59 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T21:00:59.126 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:59 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T21:00:59.126 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:59 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T21:00:59.126 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:59 smithi038 ceph-mon[26374]: Manager daemon smithi038.lxfgut is now available 2024-04-22T21:00:59.126 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:59 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:00:59.126 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:59 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:00:59.126 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:59 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:00:59.126 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:59 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:00:59.126 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:00:59 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:01:00.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:00 smithi038 ceph-mon[26374]: Found migration_current of "None". Setting to last migration. 2024-04-22T21:01:00.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:00 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.lxfgut/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:01:00.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:00 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.lxfgut/trash_purge_schedule"}]: dispatch 2024-04-22T21:01:00.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:00 smithi038 ceph-mon[26374]: mgrmap e7: smithi038.lxfgut(active, since 1.00539s) 2024-04-22T21:01:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-22T21:01:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-22T21:01:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T21:01:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-22T21:01:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.397+0000 7fd51d2a2700 1 Processor -- start 2024-04-22T21:01:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.397+0000 7fd51d2a2700 1 -- start start 2024-04-22T21:01:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.397+0000 7fd51d2a2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518101820 0x7fd518101c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.397+0000 7fd51d2a2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd518102170 con 0x7fd518101820 2024-04-22T21:01:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.397+0000 7fd516ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518101820 0x7fd518101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.397+0000 7fd516ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518101820 0x7fd518101c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43026/0 (socket says 172.21.15.38:43026) 2024-04-22T21:01:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.397+0000 7fd516ffd700 1 -- 172.21.15.38:0/210232500 learned_addr learned my addr 172.21.15.38:0/210232500 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.397+0000 7fd516ffd700 1 -- 172.21.15.38:0/210232500 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5181022b0 con 0x7fd518101820 2024-04-22T21:01:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.398+0000 7fd516ffd700 1 --2- 172.21.15.38:0/210232500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518101820 0x7fd518101c30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd5080098e0 tx=0x7fd508009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d981b5103f41113 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.398+0000 7fd515ffb700 1 -- 172.21.15.38:0/210232500 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd50800b8f0 con 0x7fd518101820 2024-04-22T21:01:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.398+0000 7fd515ffb700 1 -- 172.21.15.38:0/210232500 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd508011420 con 0x7fd518101820 2024-04-22T21:01:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.398+0000 7fd515ffb700 1 -- 172.21.15.38:0/210232500 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd508010540 con 0x7fd518101820 2024-04-22T21:01:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.398+0000 7fd51d2a2700 1 -- 172.21.15.38:0/210232500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518101820 msgr2=0x7fd518101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.398+0000 7fd51d2a2700 1 --2- 172.21.15.38:0/210232500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518101820 0x7fd518101c30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd5080098e0 tx=0x7fd508009250 comp rx=0 tx=0).stop 2024-04-22T21:01:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.398+0000 7fd51d2a2700 1 -- 172.21.15.38:0/210232500 shutdown_connections 2024-04-22T21:01:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.398+0000 7fd51d2a2700 1 --2- 172.21.15.38:0/210232500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518101820 0x7fd518101c30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:00.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.398+0000 7fd51d2a2700 1 -- 172.21.15.38:0/210232500 >> 172.21.15.38:0/210232500 conn(0x7fd5180fcf60 msgr2=0x7fd5180ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:00.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.399+0000 7fd51d2a2700 1 -- 172.21.15.38:0/210232500 shutdown_connections 2024-04-22T21:01:00.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.399+0000 7fd51d2a2700 1 -- 172.21.15.38:0/210232500 wait complete. 2024-04-22T21:01:00.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.399+0000 7fd51d2a2700 1 Processor -- start 2024-04-22T21:01:00.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.399+0000 7fd51d2a2700 1 -- start start 2024-04-22T21:01:00.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.399+0000 7fd51d2a2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518101820 0x7fd518111780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:00.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.399+0000 7fd51d2a2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd518111cc0 con 0x7fd518101820 2024-04-22T21:01:00.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.399+0000 7fd516ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518101820 0x7fd518111780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:00.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.400+0000 7fd516ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518101820 0x7fd518111780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43034/0 (socket says 172.21.15.38:43034) 2024-04-22T21:01:00.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.400+0000 7fd516ffd700 1 -- 172.21.15.38:0/3383183708 learned_addr learned my addr 172.21.15.38:0/3383183708 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:00.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.400+0000 7fd516ffd700 1 -- 172.21.15.38:0/3383183708 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5080094f0 con 0x7fd518101820 2024-04-22T21:01:00.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.400+0000 7fd516ffd700 1 --2- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518101820 0x7fd518111780 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd508011ef0 tx=0x7fd508009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:00.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.400+0000 7fd4ff7fe700 1 -- 172.21.15.38:0/3383183708 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd508010910 con 0x7fd518101820 2024-04-22T21:01:00.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.400+0000 7fd4ff7fe700 1 -- 172.21.15.38:0/3383183708 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd508010f00 con 0x7fd518101820 2024-04-22T21:01:00.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.400+0000 7fd51d2a2700 1 -- 172.21.15.38:0/3383183708 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd518111ec0 con 0x7fd518101820 2024-04-22T21:01:00.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.400+0000 7fd4ff7fe700 1 -- 172.21.15.38:0/3383183708 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd50801ae50 con 0x7fd518101820 2024-04-22T21:01:00.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.400+0000 7fd51d2a2700 1 -- 172.21.15.38:0/3383183708 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd518112a30 con 0x7fd518101820 2024-04-22T21:01:00.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.401+0000 7fd4ff7fe700 1 -- 172.21.15.38:0/3383183708 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 5) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fd50802d3f0 con 0x7fd518101820 2024-04-22T21:01:00.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.401+0000 7fd4ff7fe700 1 --2- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7fd500036740 0x7fd500038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:00.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.401+0000 7fd4ff7fe700 1 -- 172.21.15.38:0/3383183708 --> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd500039300 con 0x7fd500036740 2024-04-22T21:01:00.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.401+0000 7fd4ff7fe700 1 -- 172.21.15.38:0/3383183708 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd50804a590 con 0x7fd518101820 2024-04-22T21:01:00.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.401+0000 7fd5167fc700 1 -- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7fd500036740 msgr2=0x7fd500038bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2787760150 2024-04-22T21:01:00.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.401+0000 7fd5167fc700 1 --2- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7fd500036740 0x7fd500038bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T21:01:00.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.602+0000 7fd5167fc700 1 -- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7fd500036740 msgr2=0x7fd500038bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2787760150 2024-04-22T21:01:00.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:54.602+0000 7fd5167fc700 1 --2- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7fd500036740 0x7fd500038bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T21:01:00.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:55.003+0000 7fd5167fc700 1 -- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7fd500036740 msgr2=0x7fd500038bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2787760150 2024-04-22T21:01:00.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:55.003+0000 7fd5167fc700 1 --2- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7fd500036740 0x7fd500038bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T21:01:00.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:55.804+0000 7fd5167fc700 1 -- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7fd500036740 msgr2=0x7fd500038bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2787760150 2024-04-22T21:01:00.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:55.804+0000 7fd5167fc700 1 --2- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7fd500036740 0x7fd500038bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T21:01:00.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:57.405+0000 7fd5167fc700 1 -- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7fd500036740 msgr2=0x7fd500038bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2787760150 2024-04-22T21:01:00.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:57.405+0000 7fd5167fc700 1 --2- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7fd500036740 0x7fd500038bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T21:01:00.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:59.031+0000 7fd4ff7fe700 1 -- 172.21.15.38:0/3383183708 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mgrmap(e 6) v1 ==== 43143+0+0 (secure 0 0 0) 0x7fd50802d6f0 con 0x7fd518101820 2024-04-22T21:01:00.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:59.031+0000 7fd4ff7fe700 1 -- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7fd500036740 msgr2=0x7fd500038bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T21:01:00.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:00:59.031+0000 7fd4ff7fe700 1 --2- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7fd500036740 0x7fd500038bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:00.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.027+0000 7fd4ff7fe700 1 -- 172.21.15.38:0/3383183708 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 7) v1 ==== 43270+0+0 (secure 0 0 0) 0x7fd50802da70 con 0x7fd518101820 2024-04-22T21:01:00.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.028+0000 7fd4ff7fe700 1 --2- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fd500039fc0 0x7fd50003c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:00.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.028+0000 7fd4ff7fe700 1 -- 172.21.15.38:0/3383183708 --> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd500039300 con 0x7fd500039fc0 2024-04-22T21:01:00.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.028+0000 7fd5167fc700 1 --2- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fd500039fc0 0x7fd50003c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:00.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.028+0000 7fd5167fc700 1 --2- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fd500039fc0 0x7fd50003c3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0039d0 tx=0x7fd50c009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:00.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.029+0000 7fd4ff7fe700 1 -- 172.21.15.38:0/3383183708 <== mgr.14120 v2:172.21.15.38:6800/2551561659 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fd500039300 con 0x7fd500039fc0 2024-04-22T21:01:00.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.035+0000 7fd51d2a2700 1 -- 172.21.15.38:0/3383183708 --> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fd518112350 con 0x7fd500039fc0 2024-04-22T21:01:00.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.035+0000 7fd4ff7fe700 1 -- 172.21.15.38:0/3383183708 <== mgr.14120 v2:172.21.15.38:6800/2551561659 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd518112350 con 0x7fd500039fc0 2024-04-22T21:01:00.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.036+0000 7fd51d2a2700 1 -- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fd500039fc0 msgr2=0x7fd50003c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:00.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.036+0000 7fd51d2a2700 1 --2- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fd500039fc0 0x7fd50003c3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0039d0 tx=0x7fd50c009350 comp rx=0 tx=0).stop 2024-04-22T21:01:00.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.036+0000 7fd51d2a2700 1 -- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518101820 msgr2=0x7fd518111780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:00.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.036+0000 7fd51d2a2700 1 --2- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518101820 0x7fd518111780 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd508011ef0 tx=0x7fd508009250 comp rx=0 tx=0).stop 2024-04-22T21:01:00.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.036+0000 7fd51d2a2700 1 -- 172.21.15.38:0/3383183708 shutdown_connections 2024-04-22T21:01:00.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.036+0000 7fd51d2a2700 1 --2- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fd500039fc0 0x7fd50003c3a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:00.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.036+0000 7fd51d2a2700 1 --2- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518101820 0x7fd518111780 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:00.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.036+0000 7fd51d2a2700 1 --2- 172.21.15.38:0/3383183708 >> [v2:172.21.15.38:6800/2787760150,v1:172.21.15.38:6801/2787760150] conn(0x7fd500036740 0x7fd500038bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:00.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.036+0000 7fd51d2a2700 1 -- 172.21.15.38:0/3383183708 >> 172.21.15.38:0/3383183708 conn(0x7fd5180fcf60 msgr2=0x7fd5180fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:00.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.036+0000 7fd51d2a2700 1 -- 172.21.15.38:0/3383183708 shutdown_connections 2024-04-22T21:01:00.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:00.036+0000 7fd51d2a2700 1 -- 172.21.15.38:0/3383183708 wait complete. 2024-04-22T21:01:00.709 INFO:teuthology.orchestra.run.smithi038.stdout:mgr epoch 5 is available 2024-04-22T21:01:00.709 INFO:teuthology.orchestra.run.smithi038.stdout:Setting orchestrator backend to cephadm... 2024-04-22T21:01:01.298 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:01 smithi038 ceph-mon[26374]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T21:01:01.299 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:01 smithi038 ceph-mon[26374]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T21:01:01.940 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.078+0000 7fd8eb39d700 1 Processor -- start 2024-04-22T21:01:01.940 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.079+0000 7fd8eb39d700 1 -- start start 2024-04-22T21:01:01.940 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.079+0000 7fd8eb39d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8e4071220 0x7fd8e41055e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:01.940 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.079+0000 7fd8eb39d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e4071630 con 0x7fd8e4071220 2024-04-22T21:01:01.940 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.079+0000 7fd8e9139700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8e4071220 0x7fd8e41055e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:01.940 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.079+0000 7fd8e9139700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8e4071220 0x7fd8e41055e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43088/0 (socket says 172.21.15.38:43088) 2024-04-22T21:01:01.940 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.079+0000 7fd8e9139700 1 -- 172.21.15.38:0/1803671191 learned_addr learned my addr 172.21.15.38:0/1803671191 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:01.940 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.079+0000 7fd8e9139700 1 -- 172.21.15.38:0/1803671191 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e4105b20 con 0x7fd8e4071220 2024-04-22T21:01:01.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.079+0000 7fd8e9139700 1 --2- 172.21.15.38:0/1803671191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8e4071220 0x7fd8e41055e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00098e0 tx=0x7fd8e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e99ff71a64ec9b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:01.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.080+0000 7fd8dbfff700 1 -- 172.21.15.38:0/1803671191 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8e000b860 con 0x7fd8e4071220 2024-04-22T21:01:01.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.080+0000 7fd8dbfff700 1 -- 172.21.15.38:0/1803671191 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd8e000be50 con 0x7fd8e4071220 2024-04-22T21:01:01.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.080+0000 7fd8dbfff700 1 -- 172.21.15.38:0/1803671191 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8e0010380 con 0x7fd8e4071220 2024-04-22T21:01:01.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.080+0000 7fd8eb39d700 1 -- 172.21.15.38:0/1803671191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8e4071220 msgr2=0x7fd8e41055e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:01.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.080+0000 7fd8eb39d700 1 --2- 172.21.15.38:0/1803671191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8e4071220 0x7fd8e41055e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00098e0 tx=0x7fd8e0009250 comp rx=0 tx=0).stop 2024-04-22T21:01:01.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.081+0000 7fd8eb39d700 1 -- 172.21.15.38:0/1803671191 shutdown_connections 2024-04-22T21:01:01.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.081+0000 7fd8eb39d700 1 --2- 172.21.15.38:0/1803671191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8e4071220 0x7fd8e41055e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:01.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.081+0000 7fd8eb39d700 1 -- 172.21.15.38:0/1803671191 >> 172.21.15.38:0/1803671191 conn(0x7fd8e40fcfb0 msgr2=0x7fd8e40ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:01.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.081+0000 7fd8eb39d700 1 -- 172.21.15.38:0/1803671191 shutdown_connections 2024-04-22T21:01:01.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.081+0000 7fd8eb39d700 1 -- 172.21.15.38:0/1803671191 wait complete. 2024-04-22T21:01:01.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.082+0000 7fd8eb39d700 1 Processor -- start 2024-04-22T21:01:01.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.082+0000 7fd8eb39d700 1 -- start start 2024-04-22T21:01:01.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.082+0000 7fd8eb39d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8e4071220 0x7fd8e4111730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:01.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.082+0000 7fd8eb39d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e4111c70 con 0x7fd8e4071220 2024-04-22T21:01:01.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.082+0000 7fd8e9139700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8e4071220 0x7fd8e4111730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:01.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.082+0000 7fd8e9139700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8e4071220 0x7fd8e4111730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43092/0 (socket says 172.21.15.38:43092) 2024-04-22T21:01:01.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.082+0000 7fd8e9139700 1 -- 172.21.15.38:0/210014829 learned_addr learned my addr 172.21.15.38:0/210014829 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:01.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.083+0000 7fd8e9139700 1 -- 172.21.15.38:0/210014829 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e00094f0 con 0x7fd8e4071220 2024-04-22T21:01:01.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.083+0000 7fd8e9139700 1 --2- 172.21.15.38:0/210014829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8e4071220 0x7fd8e4111730 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00090b0 tx=0x7fd8e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:01.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.083+0000 7fd8d9ffb700 1 -- 172.21.15.38:0/210014829 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8e0010760 con 0x7fd8e4071220 2024-04-22T21:01:01.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.083+0000 7fd8d9ffb700 1 -- 172.21.15.38:0/210014829 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd8e0010d50 con 0x7fd8e4071220 2024-04-22T21:01:01.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.083+0000 7fd8eb39d700 1 -- 172.21.15.38:0/210014829 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e4111e70 con 0x7fd8e4071220 2024-04-22T21:01:01.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.083+0000 7fd8d9ffb700 1 -- 172.21.15.38:0/210014829 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8e001ace0 con 0x7fd8e4071220 2024-04-22T21:01:01.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.083+0000 7fd8eb39d700 1 -- 172.21.15.38:0/210014829 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e41129e0 con 0x7fd8e4071220 2024-04-22T21:01:01.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.084+0000 7fd8d9ffb700 1 -- 172.21.15.38:0/210014829 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fd8e002d3f0 con 0x7fd8e4071220 2024-04-22T21:01:01.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.084+0000 7fd8d9ffb700 1 --2- 172.21.15.38:0/210014829 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fd8d00367e0 0x7fd8d0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:01.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.084+0000 7fd8d9ffb700 1 -- 172.21.15.38:0/210014829 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd8e004a510 con 0x7fd8e4071220 2024-04-22T21:01:01.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.084+0000 7fd8eb39d700 1 -- 172.21.15.38:0/210014829 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e405f5f0 con 0x7fd8e4071220 2024-04-22T21:01:01.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.085+0000 7fd8e8938700 1 --2- 172.21.15.38:0/210014829 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fd8d00367e0 0x7fd8d0038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:01.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.085+0000 7fd8e8938700 1 --2- 172.21.15.38:0/210014829 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fd8d00367e0 0x7fd8d0038c90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4009990 tx=0x7fd8d4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:01.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.093+0000 7fd8d9ffb700 1 -- 172.21.15.38:0/210014829 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd8e002b8c0 con 0x7fd8e4071220 2024-04-22T21:01:01.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.302+0000 7fd8eb39d700 1 -- 172.21.15.38:0/210014829 --> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8e40fe3a0 con 0x7fd8d00367e0 2024-04-22T21:01:01.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.311+0000 7fd8d9ffb700 1 -- 172.21.15.38:0/210014829 <== mgr.14120 v2:172.21.15.38:6800/2551561659 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd8e40fe3a0 con 0x7fd8d00367e0 2024-04-22T21:01:01.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.314+0000 7fd8eb39d700 1 -- 172.21.15.38:0/210014829 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fd8d00367e0 msgr2=0x7fd8d0038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:01.944 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.314+0000 7fd8eb39d700 1 --2- 172.21.15.38:0/210014829 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fd8d00367e0 0x7fd8d0038c90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4009990 tx=0x7fd8d4006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:01.944 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.314+0000 7fd8eb39d700 1 -- 172.21.15.38:0/210014829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8e4071220 msgr2=0x7fd8e4111730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:01.944 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.314+0000 7fd8eb39d700 1 --2- 172.21.15.38:0/210014829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8e4071220 0x7fd8e4111730 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00090b0 tx=0x7fd8e0009250 comp rx=0 tx=0).stop 2024-04-22T21:01:01.944 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.314+0000 7fd8eb39d700 1 -- 172.21.15.38:0/210014829 shutdown_connections 2024-04-22T21:01:01.944 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.314+0000 7fd8eb39d700 1 --2- 172.21.15.38:0/210014829 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fd8d00367e0 0x7fd8d0038c90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:01.944 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.314+0000 7fd8eb39d700 1 --2- 172.21.15.38:0/210014829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8e4071220 0x7fd8e4111730 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:01.944 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.314+0000 7fd8eb39d700 1 -- 172.21.15.38:0/210014829 >> 172.21.15.38:0/210014829 conn(0x7fd8e40fcfb0 msgr2=0x7fd8e40fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:01.944 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.314+0000 7fd8eb39d700 1 -- 172.21.15.38:0/210014829 shutdown_connections 2024-04-22T21:01:01.944 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:01.314+0000 7fd8eb39d700 1 -- 172.21.15.38:0/210014829 wait complete. 2024-04-22T21:01:02.188 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:02 smithi038 ceph-mon[26374]: mgrmap e8: smithi038.lxfgut(active, since 2s) 2024-04-22T21:01:02.188 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:02 smithi038 ceph-mon[26374]: [22/Apr/2024:21:01:01] ENGINE Bus STARTING 2024-04-22T21:01:02.189 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:02 smithi038 ceph-mon[26374]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:01:02.189 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:02 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:02.189 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:02 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:01:02.189 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:02 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:02.189 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:02 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:01:03.160 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:03 smithi038 ceph-mon[26374]: [22/Apr/2024:21:01:01] ENGINE Serving on https://172.21.15.38:7150 2024-04-22T21:01:03.160 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:03 smithi038 ceph-mon[26374]: [22/Apr/2024:21:01:01] ENGINE Bus STARTED 2024-04-22T21:01:03.168 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-22T21:01:03.168 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.400+0000 7f465e0c1700 1 Processor -- start 2024-04-22T21:01:03.168 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.400+0000 7f465e0c1700 1 -- start start 2024-04-22T21:01:03.168 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.400+0000 7f465e0c1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4658102960 0x7f4658102d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:03.168 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.400+0000 7f465e0c1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46581032b0 con 0x7f4658102960 2024-04-22T21:01:03.168 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.400+0000 7f46577fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4658102960 0x7f4658102d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:03.168 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.400+0000 7f46577fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4658102960 0x7f4658102d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43102/0 (socket says 172.21.15.38:43102) 2024-04-22T21:01:03.168 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.400+0000 7f46577fe700 1 -- 172.21.15.38:0/3161181786 learned_addr learned my addr 172.21.15.38:0/3161181786 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.401+0000 7f46577fe700 1 -- 172.21.15.38:0/3161181786 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46581033f0 con 0x7f4658102960 2024-04-22T21:01:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.401+0000 7f46577fe700 1 --2- 172.21.15.38:0/3161181786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4658102960 0x7f4658102d70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4648009e50 tx=0x7f4648009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0197603cece64c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.401+0000 7f46567fc700 1 -- 172.21.15.38:0/3161181786 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46480092e0 con 0x7f4658102960 2024-04-22T21:01:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.401+0000 7f46567fc700 1 -- 172.21.15.38:0/3161181786 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4648012430 con 0x7f4658102960 2024-04-22T21:01:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.401+0000 7f46567fc700 1 -- 172.21.15.38:0/3161181786 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46480075c0 con 0x7f4658102960 2024-04-22T21:01:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.402+0000 7f465e0c1700 1 -- 172.21.15.38:0/3161181786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4658102960 msgr2=0x7f4658102d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.402+0000 7f465e0c1700 1 --2- 172.21.15.38:0/3161181786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4658102960 0x7f4658102d70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4648009e50 tx=0x7f4648009840 comp rx=0 tx=0).stop 2024-04-22T21:01:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.402+0000 7f465e0c1700 1 -- 172.21.15.38:0/3161181786 shutdown_connections 2024-04-22T21:01:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.402+0000 7f465e0c1700 1 --2- 172.21.15.38:0/3161181786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4658102960 0x7f4658102d70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.402+0000 7f465e0c1700 1 -- 172.21.15.38:0/3161181786 >> 172.21.15.38:0/3161181786 conn(0x7f46580fe0e0 msgr2=0x7f46581004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.402+0000 7f465e0c1700 1 -- 172.21.15.38:0/3161181786 shutdown_connections 2024-04-22T21:01:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.402+0000 7f465e0c1700 1 -- 172.21.15.38:0/3161181786 wait complete. 2024-04-22T21:01:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.402+0000 7f465e0c1700 1 Processor -- start 2024-04-22T21:01:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.403+0000 7f465e0c1700 1 -- start start 2024-04-22T21:01:03.170 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.403+0000 7f465e0c1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4658102960 0x7f4658078c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:03.170 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.403+0000 7f465e0c1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4658079190 con 0x7f4658102960 2024-04-22T21:01:03.170 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.403+0000 7f46577fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4658102960 0x7f4658078c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:03.170 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.403+0000 7f46577fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4658102960 0x7f4658078c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43106/0 (socket says 172.21.15.38:43106) 2024-04-22T21:01:03.170 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.403+0000 7f46577fe700 1 -- 172.21.15.38:0/3518358450 learned_addr learned my addr 172.21.15.38:0/3518358450 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:03.170 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.403+0000 7f46577fe700 1 -- 172.21.15.38:0/3518358450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4648009ae0 con 0x7f4658102960 2024-04-22T21:01:03.170 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.403+0000 7f46577fe700 1 --2- 172.21.15.38:0/3518358450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4658102960 0x7f4658078c50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4648012da0 tx=0x7f4648009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:03.170 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.403+0000 7f464ffff700 1 -- 172.21.15.38:0/3518358450 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46480092e0 con 0x7f4658102960 2024-04-22T21:01:03.170 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.403+0000 7f464ffff700 1 -- 172.21.15.38:0/3518358450 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4648007d10 con 0x7f4658102960 2024-04-22T21:01:03.170 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.403+0000 7f465e0c1700 1 -- 172.21.15.38:0/3518358450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4658077260 con 0x7f4658102960 2024-04-22T21:01:03.170 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.403+0000 7f464ffff700 1 -- 172.21.15.38:0/3518358450 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f464801c8c0 con 0x7f4658102960 2024-04-22T21:01:03.170 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.403+0000 7f465e0c1700 1 -- 172.21.15.38:0/3518358450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4658077750 con 0x7f4658102960 2024-04-22T21:01:03.170 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.404+0000 7f464ffff700 1 -- 172.21.15.38:0/3518358450 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f4648019070 con 0x7f4658102960 2024-04-22T21:01:03.171 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.404+0000 7f464ffff700 1 --2- 172.21.15.38:0/3518358450 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f4644036790 0x7f4644038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:03.171 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.404+0000 7f464ffff700 1 -- 172.21.15.38:0/3518358450 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f4648049f20 con 0x7f4658102960 2024-04-22T21:01:03.171 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.405+0000 7f4656ffd700 1 --2- 172.21.15.38:0/3518358450 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f4644036790 0x7f4644038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:03.171 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.405+0000 7f465e0c1700 1 -- 172.21.15.38:0/3518358450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f465804bed0 con 0x7f4658102960 2024-04-22T21:01:03.171 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.405+0000 7f4656ffd700 1 --2- 172.21.15.38:0/3518358450 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f4644036790 0x7f4644038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4640009990 tx=0x7f4640006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:03.171 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.410+0000 7f464ffff700 1 -- 172.21.15.38:0/3518358450 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f4648020080 con 0x7f4658102960 2024-04-22T21:01:03.171 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.605+0000 7f465e0c1700 1 -- 172.21.15.38:0/3518358450 --> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f4658077c40 con 0x7f4644036790 2024-04-22T21:01:03.171 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.606+0000 7f464ffff700 1 -- 172.21.15.38:0/3518358450 <== mgr.14120 v2:172.21.15.38:6800/2551561659 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f4658077c40 con 0x7f4644036790 2024-04-22T21:01:03.171 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.608+0000 7f465e0c1700 1 -- 172.21.15.38:0/3518358450 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f4644036790 msgr2=0x7f4644038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:03.171 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.608+0000 7f465e0c1700 1 --2- 172.21.15.38:0/3518358450 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f4644036790 0x7f4644038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4640009990 tx=0x7f4640006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:03.171 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.608+0000 7f465e0c1700 1 -- 172.21.15.38:0/3518358450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4658102960 msgr2=0x7f4658078c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:03.171 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.608+0000 7f465e0c1700 1 --2- 172.21.15.38:0/3518358450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4658102960 0x7f4658078c50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4648012da0 tx=0x7f4648009840 comp rx=0 tx=0).stop 2024-04-22T21:01:03.171 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.608+0000 7f465e0c1700 1 -- 172.21.15.38:0/3518358450 shutdown_connections 2024-04-22T21:01:03.171 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.608+0000 7f465e0c1700 1 --2- 172.21.15.38:0/3518358450 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f4644036790 0x7f4644038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:03.172 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.608+0000 7f465e0c1700 1 --2- 172.21.15.38:0/3518358450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4658102960 0x7f4658078c50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:03.172 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.608+0000 7f465e0c1700 1 -- 172.21.15.38:0/3518358450 >> 172.21.15.38:0/3518358450 conn(0x7f46580fe0e0 msgr2=0x7f46580fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:03.172 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.608+0000 7f465e0c1700 1 -- 172.21.15.38:0/3518358450 shutdown_connections 2024-04-22T21:01:03.172 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:02.608+0000 7f465e0c1700 1 -- 172.21.15.38:0/3518358450 wait complete. 2024-04-22T21:01:03.172 INFO:teuthology.orchestra.run.smithi038.stdout:Generating ssh key... 2024-04-22T21:01:04.339 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:04 smithi038 ceph-mon[26374]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:01:04.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.592+0000 7f4dbf6cc700 1 Processor -- start 2024-04-22T21:01:04.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.592+0000 7f4dbf6cc700 1 -- start start 2024-04-22T21:01:04.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.592+0000 7f4dbf6cc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db8076b40 0x7f4db8105570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:04.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.592+0000 7f4dbf6cc700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db8076f50 con 0x7f4db8076b40 2024-04-22T21:01:04.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.592+0000 7f4dbd468700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db8076b40 0x7f4db8105570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:04.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.592+0000 7f4dbd468700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db8076b40 0x7f4db8105570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39056/0 (socket says 172.21.15.38:39056) 2024-04-22T21:01:04.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.592+0000 7f4dbd468700 1 -- 172.21.15.38:0/631964488 learned_addr learned my addr 172.21.15.38:0/631964488 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:04.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.593+0000 7f4dbd468700 1 -- 172.21.15.38:0/631964488 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db8105ab0 con 0x7f4db8076b40 2024-04-22T21:01:04.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.593+0000 7f4dbd468700 1 --2- 172.21.15.38:0/631964488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db8076b40 0x7f4db8105570 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4db40098e0 tx=0x7f4db4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d459e70d638bde server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:04.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.593+0000 7f4daffff700 1 -- 172.21.15.38:0/631964488 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4db400b8f0 con 0x7f4db8076b40 2024-04-22T21:01:04.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.593+0000 7f4daffff700 1 -- 172.21.15.38:0/631964488 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4db4011420 con 0x7f4db8076b40 2024-04-22T21:01:04.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.593+0000 7f4daffff700 1 -- 172.21.15.38:0/631964488 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4db4010540 con 0x7f4db8076b40 2024-04-22T21:01:04.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.594+0000 7f4dbf6cc700 1 -- 172.21.15.38:0/631964488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db8076b40 msgr2=0x7f4db8105570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:04.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.594+0000 7f4dbf6cc700 1 --2- 172.21.15.38:0/631964488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db8076b40 0x7f4db8105570 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4db40098e0 tx=0x7f4db4009250 comp rx=0 tx=0).stop 2024-04-22T21:01:04.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.594+0000 7f4dbf6cc700 1 -- 172.21.15.38:0/631964488 shutdown_connections 2024-04-22T21:01:04.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.594+0000 7f4dbf6cc700 1 --2- 172.21.15.38:0/631964488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db8076b40 0x7f4db8105570 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:04.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.594+0000 7f4dbf6cc700 1 -- 172.21.15.38:0/631964488 >> 172.21.15.38:0/631964488 conn(0x7f4db80fcfe0 msgr2=0x7f4db80ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:04.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.594+0000 7f4dbf6cc700 1 -- 172.21.15.38:0/631964488 shutdown_connections 2024-04-22T21:01:04.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.594+0000 7f4dbf6cc700 1 -- 172.21.15.38:0/631964488 wait complete. 2024-04-22T21:01:04.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.594+0000 7f4dbf6cc700 1 Processor -- start 2024-04-22T21:01:04.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.594+0000 7f4dbf6cc700 1 -- start start 2024-04-22T21:01:04.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.595+0000 7f4dbf6cc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db8076b40 0x7f4db8102ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:04.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.595+0000 7f4dbf6cc700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db8103400 con 0x7f4db8076b40 2024-04-22T21:01:04.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.595+0000 7f4dbd468700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db8076b40 0x7f4db8102ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:04.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.595+0000 7f4dbd468700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db8076b40 0x7f4db8102ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39070/0 (socket says 172.21.15.38:39070) 2024-04-22T21:01:04.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.595+0000 7f4dbd468700 1 -- 172.21.15.38:0/3754209493 learned_addr learned my addr 172.21.15.38:0/3754209493 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:04.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.595+0000 7f4dbd468700 1 -- 172.21.15.38:0/3754209493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db40094f0 con 0x7f4db8076b40 2024-04-22T21:01:04.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.595+0000 7f4dbd468700 1 --2- 172.21.15.38:0/3754209493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db8076b40 0x7f4db8102ec0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4db40094c0 tx=0x7f4db4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:04.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.595+0000 7f4dadffb700 1 -- 172.21.15.38:0/3754209493 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4db4010800 con 0x7f4db8076b40 2024-04-22T21:01:04.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.595+0000 7f4dadffb700 1 -- 172.21.15.38:0/3754209493 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4db4010df0 con 0x7f4db8076b40 2024-04-22T21:01:04.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.595+0000 7f4dbf6cc700 1 -- 172.21.15.38:0/3754209493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db8101470 con 0x7f4db8076b40 2024-04-22T21:01:04.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.595+0000 7f4dadffb700 1 -- 172.21.15.38:0/3754209493 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4db401add0 con 0x7f4db8076b40 2024-04-22T21:01:04.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.596+0000 7f4dbf6cc700 1 -- 172.21.15.38:0/3754209493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db8101960 con 0x7f4db8076b40 2024-04-22T21:01:04.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.596+0000 7f4dadffb700 1 -- 172.21.15.38:0/3754209493 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f4db402d3f0 con 0x7f4db8076b40 2024-04-22T21:01:04.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.597+0000 7f4dbf6cc700 1 -- 172.21.15.38:0/3754209493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db804bed0 con 0x7f4db8076b40 2024-04-22T21:01:04.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.597+0000 7f4dadffb700 1 --2- 172.21.15.38:0/3754209493 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f4da40367e0 0x7f4da4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:04.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.597+0000 7f4dadffb700 1 -- 172.21.15.38:0/3754209493 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f4db404a480 con 0x7f4db8076b40 2024-04-22T21:01:04.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.597+0000 7f4dbcc67700 1 --2- 172.21.15.38:0/3754209493 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f4da40367e0 0x7f4da4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:04.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.597+0000 7f4dbcc67700 1 --2- 172.21.15.38:0/3754209493 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f4da40367e0 0x7f4da4038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4da8009990 tx=0x7f4da8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:04.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.601+0000 7f4dadffb700 1 -- 172.21.15.38:0/3754209493 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f4db401a650 con 0x7f4db8076b40 2024-04-22T21:01:04.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:03.798+0000 7f4dbf6cc700 1 -- 172.21.15.38:0/3754209493 --> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f4db810b5d0 con 0x7f4da40367e0 2024-04-22T21:01:04.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:04.330+0000 7f4dadffb700 1 -- 172.21.15.38:0/3754209493 <== mgr.14120 v2:172.21.15.38:6800/2551561659 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4db810b5d0 con 0x7f4da40367e0 2024-04-22T21:01:04.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:04.332+0000 7f4dbf6cc700 1 -- 172.21.15.38:0/3754209493 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f4da40367e0 msgr2=0x7f4da4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:04.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:04.332+0000 7f4dbf6cc700 1 --2- 172.21.15.38:0/3754209493 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f4da40367e0 0x7f4da4038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4da8009990 tx=0x7f4da8006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:04.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:04.332+0000 7f4dbf6cc700 1 -- 172.21.15.38:0/3754209493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db8076b40 msgr2=0x7f4db8102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:04.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:04.332+0000 7f4dbf6cc700 1 --2- 172.21.15.38:0/3754209493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db8076b40 0x7f4db8102ec0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4db40094c0 tx=0x7f4db4009250 comp rx=0 tx=0).stop 2024-04-22T21:01:04.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:04.332+0000 7f4dbf6cc700 1 -- 172.21.15.38:0/3754209493 shutdown_connections 2024-04-22T21:01:04.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:04.332+0000 7f4dbf6cc700 1 --2- 172.21.15.38:0/3754209493 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f4da40367e0 0x7f4da4038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:04.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:04.332+0000 7f4dbf6cc700 1 --2- 172.21.15.38:0/3754209493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db8076b40 0x7f4db8102ec0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:04.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:04.332+0000 7f4dbf6cc700 1 -- 172.21.15.38:0/3754209493 >> 172.21.15.38:0/3754209493 conn(0x7f4db80fcfe0 msgr2=0x7f4db80fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:04.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:04.332+0000 7f4dbf6cc700 1 -- 172.21.15.38:0/3754209493 shutdown_connections 2024-04-22T21:01:04.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:04.332+0000 7f4dbf6cc700 1 -- 172.21.15.38:0/3754209493 wait complete. 2024-04-22T21:01:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:05 smithi038 ceph-mon[26374]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:01:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:05 smithi038 ceph-mon[26374]: Generating ssh key... 2024-04-22T21:01:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:05 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:05 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:06.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwKQaHt7Hq/TAdPcNj15MTJBeTmwjU51OPBip51SAd/PK7M7x8PwFUVGlWm+dh/qoSbqu5uTYyKv326j4g1ithmDBuYUH6tFgh24ICByEpcPVppSJRZ7ZLgU6M4IlDZaM9B3AEjjjyIbtM1Ij3e2r0OGLyvG3Gvvrys1GeYygCl5k41LiBI/8QTmClGQF5F/rb06ExAt1ZgIrAzJeNoDynx6tZpVtScTE8xmi56Gt+f4SE3cSooxLPUS2ksFv/08fTZY5JvkS0QA/A9iC6fda0OFWNn9+nzdK7CPOm8L+Tyojh9WnytzQzbAS/BBfsQZHs6y9Hv7BD1n4CNk/6yquq/8xKu4FnZ+gnU4kL8sBRnTOB7YieIQ6jFTPtBo0Im5PFAQYK3X/SjS1SudYXt53wMuv/cNRnMXsCxHLSo6k+6svzxgbTlb7dSs2C2caNHLkwVU/7Gc1pJqSZ7lLnG8CxQniOjiyc+eq1ZGnOquYM1F4vjfGvJ5j+Rp27bEJDY2k= ceph-3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T21:01:06.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.316+0000 7f3f2a7cd700 1 Processor -- start 2024-04-22T21:01:06.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.316+0000 7f3f2a7cd700 1 -- start start 2024-04-22T21:01:06.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.316+0000 7f3f2a7cd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f24076b20 0x7f3f241055a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:06.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.316+0000 7f3f2a7cd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f24076f30 con 0x7f3f24076b20 2024-04-22T21:01:06.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.316+0000 7f3f23fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f24076b20 0x7f3f241055a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:06.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.316+0000 7f3f23fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f24076b20 0x7f3f241055a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39082/0 (socket says 172.21.15.38:39082) 2024-04-22T21:01:06.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.316+0000 7f3f23fff700 1 -- 172.21.15.38:0/1463498698 learned_addr learned my addr 172.21.15.38:0/1463498698 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:06.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.316+0000 7f3f23fff700 1 -- 172.21.15.38:0/1463498698 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f24105ae0 con 0x7f3f24076b20 2024-04-22T21:01:06.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.317+0000 7f3f23fff700 1 --2- 172.21.15.38:0/1463498698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f24076b20 0x7f3f241055a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3f140098e0 tx=0x7f3f14009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=803471bb655de01a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:06.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.317+0000 7f3f22ffd700 1 -- 172.21.15.38:0/1463498698 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f1400b8f0 con 0x7f3f24076b20 2024-04-22T21:01:06.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.317+0000 7f3f22ffd700 1 -- 172.21.15.38:0/1463498698 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3f14011420 con 0x7f3f24076b20 2024-04-22T21:01:06.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.317+0000 7f3f22ffd700 1 -- 172.21.15.38:0/1463498698 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f14010540 con 0x7f3f24076b20 2024-04-22T21:01:06.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.317+0000 7f3f2a7cd700 1 -- 172.21.15.38:0/1463498698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f24076b20 msgr2=0x7f3f241055a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:06.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.317+0000 7f3f2a7cd700 1 --2- 172.21.15.38:0/1463498698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f24076b20 0x7f3f241055a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3f140098e0 tx=0x7f3f14009250 comp rx=0 tx=0).stop 2024-04-22T21:01:06.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.317+0000 7f3f2a7cd700 1 -- 172.21.15.38:0/1463498698 shutdown_connections 2024-04-22T21:01:06.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.317+0000 7f3f2a7cd700 1 --2- 172.21.15.38:0/1463498698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f24076b20 0x7f3f241055a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:06.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.317+0000 7f3f2a7cd700 1 -- 172.21.15.38:0/1463498698 >> 172.21.15.38:0/1463498698 conn(0x7f3f240fd010 msgr2=0x7f3f240ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:06.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.317+0000 7f3f2a7cd700 1 -- 172.21.15.38:0/1463498698 shutdown_connections 2024-04-22T21:01:06.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.318+0000 7f3f2a7cd700 1 -- 172.21.15.38:0/1463498698 wait complete. 2024-04-22T21:01:06.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.318+0000 7f3f2a7cd700 1 Processor -- start 2024-04-22T21:01:06.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.318+0000 7f3f2a7cd700 1 -- start start 2024-04-22T21:01:06.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.319+0000 7f3f2a7cd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f24076b20 0x7f3f24102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:06.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.319+0000 7f3f2a7cd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f24103430 con 0x7f3f24076b20 2024-04-22T21:01:06.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.319+0000 7f3f23fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f24076b20 0x7f3f24102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:06.125 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.319+0000 7f3f23fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f24076b20 0x7f3f24102ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39092/0 (socket says 172.21.15.38:39092) 2024-04-22T21:01:06.125 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.319+0000 7f3f23fff700 1 -- 172.21.15.38:0/3652160370 learned_addr learned my addr 172.21.15.38:0/3652160370 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:06.125 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.319+0000 7f3f23fff700 1 -- 172.21.15.38:0/3652160370 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f140094f0 con 0x7f3f24076b20 2024-04-22T21:01:06.125 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.319+0000 7f3f23fff700 1 --2- 172.21.15.38:0/3652160370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f24076b20 0x7f3f24102ef0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f140094c0 tx=0x7f3f14009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:06.125 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.319+0000 7f3f20ff9700 1 -- 172.21.15.38:0/3652160370 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f14010800 con 0x7f3f24076b20 2024-04-22T21:01:06.125 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.319+0000 7f3f20ff9700 1 -- 172.21.15.38:0/3652160370 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3f14010df0 con 0x7f3f24076b20 2024-04-22T21:01:06.125 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.319+0000 7f3f20ff9700 1 -- 172.21.15.38:0/3652160370 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f1401add0 con 0x7f3f24076b20 2024-04-22T21:01:06.125 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.319+0000 7f3f2a7cd700 1 -- 172.21.15.38:0/3652160370 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f24101500 con 0x7f3f24076b20 2024-04-22T21:01:06.125 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.319+0000 7f3f2a7cd700 1 -- 172.21.15.38:0/3652160370 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f241019f0 con 0x7f3f24076b20 2024-04-22T21:01:06.125 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.320+0000 7f3f20ff9700 1 -- 172.21.15.38:0/3652160370 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f3f14010960 con 0x7f3f24076b20 2024-04-22T21:01:06.125 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.320+0000 7f3f20ff9700 1 --2- 172.21.15.38:0/3652160370 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f3f100367e0 0x7f3f10038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:06.125 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.320+0000 7f3f2a7cd700 1 -- 172.21.15.38:0/3652160370 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f2404bed0 con 0x7f3f24076b20 2024-04-22T21:01:06.125 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.320+0000 7f3f20ff9700 1 -- 172.21.15.38:0/3652160370 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3f1404a470 con 0x7f3f24076b20 2024-04-22T21:01:06.125 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.321+0000 7f3f237fe700 1 --2- 172.21.15.38:0/3652160370 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f3f100367e0 0x7f3f10038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:06.126 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.321+0000 7f3f237fe700 1 --2- 172.21.15.38:0/3652160370 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f3f100367e0 0x7f3f10038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c009990 tx=0x7f3f0c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:06.126 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.326+0000 7f3f20ff9700 1 -- 172.21.15.38:0/3652160370 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f3f1401a430 con 0x7f3f24076b20 2024-04-22T21:01:06.126 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.522+0000 7f3f2a7cd700 1 -- 172.21.15.38:0/3652160370 --> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f2410b5a0 con 0x7f3f100367e0 2024-04-22T21:01:06.126 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.523+0000 7f3f20ff9700 1 -- 172.21.15.38:0/3652160370 <== mgr.14120 v2:172.21.15.38:6800/2551561659 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f3f2410b5a0 con 0x7f3f100367e0 2024-04-22T21:01:06.126 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.524+0000 7f3f2a7cd700 1 -- 172.21.15.38:0/3652160370 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f3f100367e0 msgr2=0x7f3f10038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:06.126 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.524+0000 7f3f2a7cd700 1 --2- 172.21.15.38:0/3652160370 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f3f100367e0 0x7f3f10038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c009990 tx=0x7f3f0c006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:06.126 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.524+0000 7f3f2a7cd700 1 -- 172.21.15.38:0/3652160370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f24076b20 msgr2=0x7f3f24102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:06.126 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.524+0000 7f3f2a7cd700 1 --2- 172.21.15.38:0/3652160370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f24076b20 0x7f3f24102ef0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f140094c0 tx=0x7f3f14009250 comp rx=0 tx=0).stop 2024-04-22T21:01:06.126 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.524+0000 7f3f2a7cd700 1 -- 172.21.15.38:0/3652160370 shutdown_connections 2024-04-22T21:01:06.126 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.524+0000 7f3f2a7cd700 1 --2- 172.21.15.38:0/3652160370 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f3f100367e0 0x7f3f10038c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:06.126 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.524+0000 7f3f2a7cd700 1 --2- 172.21.15.38:0/3652160370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f24076b20 0x7f3f24102ef0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:06.126 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.524+0000 7f3f2a7cd700 1 -- 172.21.15.38:0/3652160370 >> 172.21.15.38:0/3652160370 conn(0x7f3f240fd010 msgr2=0x7f3f240fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:06.126 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.525+0000 7f3f2a7cd700 1 -- 172.21.15.38:0/3652160370 shutdown_connections 2024-04-22T21:01:06.126 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:05.525+0000 7f3f2a7cd700 1 -- 172.21.15.38:0/3652160370 wait complete. 2024-04-22T21:01:06.127 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-22T21:01:06.127 INFO:teuthology.orchestra.run.smithi038.stdout:Adding key to root@localhost authorized_keys... 2024-04-22T21:01:06.127 INFO:teuthology.orchestra.run.smithi038.stdout:Adding host smithi038... 2024-04-22T21:01:06.510 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:06 smithi038 ceph-mon[26374]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:01:07.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:07 smithi038 ceph-mon[26374]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi038", "addr": "172.21.15.38", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:01:07.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:07 smithi038 ceph-mon[26374]: Deploying cephadm binary to smithi038 2024-04-22T21:01:09.289 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Added host 'smithi038' with addr '172.21.15.38' 2024-04-22T21:01:09.289 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.553+0000 7f1815a12700 1 Processor -- start 2024-04-22T21:01:09.289 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.553+0000 7f1815a12700 1 -- start start 2024-04-22T21:01:09.289 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.553+0000 7f1815a12700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1810071300 0x7f1810105500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:09.290 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.553+0000 7f1815a12700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1810071710 con 0x7f1810071300 2024-04-22T21:01:09.290 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.553+0000 7f180effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1810071300 0x7f1810105500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:09.290 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.553+0000 7f180effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1810071300 0x7f1810105500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39094/0 (socket says 172.21.15.38:39094) 2024-04-22T21:01:09.290 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.553+0000 7f180effd700 1 -- 172.21.15.38:0/1494456880 learned_addr learned my addr 172.21.15.38:0/1494456880 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:09.290 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.554+0000 7f180effd700 1 -- 172.21.15.38:0/1494456880 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1810105a40 con 0x7f1810071300 2024-04-22T21:01:09.290 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.554+0000 7f180effd700 1 --2- 172.21.15.38:0/1494456880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1810071300 0x7f1810105500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f18000098e0 tx=0x7f1800009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be670af5c467dd9b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:09.290 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.554+0000 7f180dffb700 1 -- 172.21.15.38:0/1494456880 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f180000b860 con 0x7f1810071300 2024-04-22T21:01:09.290 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.554+0000 7f180dffb700 1 -- 172.21.15.38:0/1494456880 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f180000be50 con 0x7f1810071300 2024-04-22T21:01:09.290 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.554+0000 7f180dffb700 1 -- 172.21.15.38:0/1494456880 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1800010380 con 0x7f1810071300 2024-04-22T21:01:09.290 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.554+0000 7f1815a12700 1 -- 172.21.15.38:0/1494456880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1810071300 msgr2=0x7f1810105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:09.290 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.554+0000 7f1815a12700 1 --2- 172.21.15.38:0/1494456880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1810071300 0x7f1810105500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f18000098e0 tx=0x7f1800009250 comp rx=0 tx=0).stop 2024-04-22T21:01:09.290 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.555+0000 7f1815a12700 1 -- 172.21.15.38:0/1494456880 shutdown_connections 2024-04-22T21:01:09.290 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.555+0000 7f1815a12700 1 --2- 172.21.15.38:0/1494456880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1810071300 0x7f1810105500 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:09.291 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.555+0000 7f1815a12700 1 -- 172.21.15.38:0/1494456880 >> 172.21.15.38:0/1494456880 conn(0x7f18100fcfb0 msgr2=0x7f18100ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:09.291 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.555+0000 7f1815a12700 1 -- 172.21.15.38:0/1494456880 shutdown_connections 2024-04-22T21:01:09.291 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.555+0000 7f1815a12700 1 -- 172.21.15.38:0/1494456880 wait complete. 2024-04-22T21:01:09.291 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.555+0000 7f1815a12700 1 Processor -- start 2024-04-22T21:01:09.291 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.555+0000 7f1815a12700 1 -- start start 2024-04-22T21:01:09.291 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.556+0000 7f1815a12700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1810071300 0x7f1810111780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:09.291 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.556+0000 7f1815a12700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1810111cc0 con 0x7f1810071300 2024-04-22T21:01:09.291 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.556+0000 7f180effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1810071300 0x7f1810111780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:09.291 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.556+0000 7f180effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1810071300 0x7f1810111780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39100/0 (socket says 172.21.15.38:39100) 2024-04-22T21:01:09.291 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.556+0000 7f180effd700 1 -- 172.21.15.38:0/2842570888 learned_addr learned my addr 172.21.15.38:0/2842570888 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:09.291 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.556+0000 7f180effd700 1 -- 172.21.15.38:0/2842570888 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18000094f0 con 0x7f1810071300 2024-04-22T21:01:09.291 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.556+0000 7f180effd700 1 --2- 172.21.15.38:0/2842570888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1810071300 0x7f1810111780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f18000090b0 tx=0x7f1800009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:09.291 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.556+0000 7f17f77fe700 1 -- 172.21.15.38:0/2842570888 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1800010760 con 0x7f1810071300 2024-04-22T21:01:09.291 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.556+0000 7f17f77fe700 1 -- 172.21.15.38:0/2842570888 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1800010d50 con 0x7f1810071300 2024-04-22T21:01:09.292 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.556+0000 7f1815a12700 1 -- 172.21.15.38:0/2842570888 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1810111ec0 con 0x7f1810071300 2024-04-22T21:01:09.292 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.556+0000 7f17f77fe700 1 -- 172.21.15.38:0/2842570888 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f180001ace0 con 0x7f1810071300 2024-04-22T21:01:09.292 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.556+0000 7f1815a12700 1 -- 172.21.15.38:0/2842570888 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1810112a30 con 0x7f1810071300 2024-04-22T21:01:09.292 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.557+0000 7f17f77fe700 1 -- 172.21.15.38:0/2842570888 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f180002d3f0 con 0x7f1810071300 2024-04-22T21:01:09.292 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.557+0000 7f1815a12700 1 -- 172.21.15.38:0/2842570888 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181005f5f0 con 0x7f1810071300 2024-04-22T21:01:09.292 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.557+0000 7f17f77fe700 1 --2- 172.21.15.38:0/2842570888 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f17f80367e0 0x7f17f8038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:09.292 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.558+0000 7f17f77fe700 1 -- 172.21.15.38:0/2842570888 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f180004a140 con 0x7f1810071300 2024-04-22T21:01:09.292 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.558+0000 7f180e7fc700 1 --2- 172.21.15.38:0/2842570888 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f17f80367e0 0x7f17f8038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:09.292 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.558+0000 7f180e7fc700 1 --2- 172.21.15.38:0/2842570888 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f17f80367e0 0x7f17f8038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1804009990 tx=0x7f1804006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:09.292 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.562+0000 7f17f77fe700 1 -- 172.21.15.38:0/2842570888 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f180002b4d0 con 0x7f1810071300 2024-04-22T21:01:09.292 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:06.761+0000 7f1815a12700 1 -- 172.21.15.38:0/2842570888 --> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi038", "addr": "172.21.15.38", "target": ["mon-mgr", ""]}) v1 -- 0x7f18100fe470 con 0x7f17f80367e0 2024-04-22T21:01:09.292 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:08.695+0000 7f17f77fe700 1 -- 172.21.15.38:0/2842570888 <== mgr.14120 v2:172.21.15.38:6800/2551561659 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f18100fe470 con 0x7f17f80367e0 2024-04-22T21:01:09.292 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:08.697+0000 7f1815a12700 1 -- 172.21.15.38:0/2842570888 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f17f80367e0 msgr2=0x7f17f8038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:09.293 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:08.697+0000 7f1815a12700 1 --2- 172.21.15.38:0/2842570888 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f17f80367e0 0x7f17f8038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1804009990 tx=0x7f1804006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:09.293 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:08.697+0000 7f1815a12700 1 -- 172.21.15.38:0/2842570888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1810071300 msgr2=0x7f1810111780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:09.293 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:08.697+0000 7f1815a12700 1 --2- 172.21.15.38:0/2842570888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1810071300 0x7f1810111780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f18000090b0 tx=0x7f1800009250 comp rx=0 tx=0).stop 2024-04-22T21:01:09.293 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:08.697+0000 7f1815a12700 1 -- 172.21.15.38:0/2842570888 shutdown_connections 2024-04-22T21:01:09.293 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:08.697+0000 7f1815a12700 1 --2- 172.21.15.38:0/2842570888 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f17f80367e0 0x7f17f8038c90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:09.293 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:08.697+0000 7f1815a12700 1 --2- 172.21.15.38:0/2842570888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1810071300 0x7f1810111780 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:09.293 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:08.697+0000 7f1815a12700 1 -- 172.21.15.38:0/2842570888 >> 172.21.15.38:0/2842570888 conn(0x7f18100fcfb0 msgr2=0x7f18100fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:09.293 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:08.697+0000 7f1815a12700 1 -- 172.21.15.38:0/2842570888 shutdown_connections 2024-04-22T21:01:09.293 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:08.697+0000 7f1815a12700 1 -- 172.21.15.38:0/2842570888 wait complete. 2024-04-22T21:01:09.293 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying mon service with default placement... 2024-04-22T21:01:09.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:09 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:09.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:09 smithi038 ceph-mon[26374]: Added host smithi038 2024-04-22T21:01:09.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:09 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:01:10.758 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-22T21:01:10.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.710+0000 7f133bd52700 1 Processor -- start 2024-04-22T21:01:10.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.710+0000 7f133bd52700 1 -- start start 2024-04-22T21:01:10.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.710+0000 7f133bd52700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1334102930 0x7f1334102d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:10.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.710+0000 7f133bd52700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1334103280 con 0x7f1334102930 2024-04-22T21:01:10.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.710+0000 7f1339aee700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1334102930 0x7f1334102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:10.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.710+0000 7f1339aee700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1334102930 0x7f1334102d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39108/0 (socket says 172.21.15.38:39108) 2024-04-22T21:01:10.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.710+0000 7f1339aee700 1 -- 172.21.15.38:0/2166379108 learned_addr learned my addr 172.21.15.38:0/2166379108 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:10.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.711+0000 7f1339aee700 1 -- 172.21.15.38:0/2166379108 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13341033c0 con 0x7f1334102930 2024-04-22T21:01:10.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.711+0000 7f1339aee700 1 --2- 172.21.15.38:0/2166379108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1334102930 0x7f1334102d40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f13300098e0 tx=0x7f1330009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bedeee71c07813ee server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:10.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.711+0000 7f1338aec700 1 -- 172.21.15.38:0/2166379108 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f133000b8f0 con 0x7f1334102930 2024-04-22T21:01:10.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.711+0000 7f1338aec700 1 -- 172.21.15.38:0/2166379108 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1330011420 con 0x7f1334102930 2024-04-22T21:01:10.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.711+0000 7f1338aec700 1 -- 172.21.15.38:0/2166379108 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1330010540 con 0x7f1334102930 2024-04-22T21:01:10.759 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.711+0000 7f133bd52700 1 -- 172.21.15.38:0/2166379108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1334102930 msgr2=0x7f1334102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:10.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.711+0000 7f133bd52700 1 --2- 172.21.15.38:0/2166379108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1334102930 0x7f1334102d40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f13300098e0 tx=0x7f1330009250 comp rx=0 tx=0).stop 2024-04-22T21:01:10.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.711+0000 7f133bd52700 1 -- 172.21.15.38:0/2166379108 shutdown_connections 2024-04-22T21:01:10.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.711+0000 7f133bd52700 1 --2- 172.21.15.38:0/2166379108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1334102930 0x7f1334102d40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:10.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.711+0000 7f133bd52700 1 -- 172.21.15.38:0/2166379108 >> 172.21.15.38:0/2166379108 conn(0x7f13340fe140 msgr2=0x7f1334100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:10.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.712+0000 7f133bd52700 1 -- 172.21.15.38:0/2166379108 shutdown_connections 2024-04-22T21:01:10.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.712+0000 7f133bd52700 1 -- 172.21.15.38:0/2166379108 wait complete. 2024-04-22T21:01:10.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.712+0000 7f133bd52700 1 Processor -- start 2024-04-22T21:01:10.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.712+0000 7f133bd52700 1 -- start start 2024-04-22T21:01:10.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.712+0000 7f133bd52700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1334102930 0x7f1334111750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:10.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.712+0000 7f133bd52700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1334111c90 con 0x7f1334102930 2024-04-22T21:01:10.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.712+0000 7f1339aee700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1334102930 0x7f1334111750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:10.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.712+0000 7f1339aee700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1334102930 0x7f1334111750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39112/0 (socket says 172.21.15.38:39112) 2024-04-22T21:01:10.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.712+0000 7f1339aee700 1 -- 172.21.15.38:0/1353156773 learned_addr learned my addr 172.21.15.38:0/1353156773 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:10.760 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.713+0000 7f1339aee700 1 -- 172.21.15.38:0/1353156773 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13300094f0 con 0x7f1334102930 2024-04-22T21:01:10.761 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.713+0000 7f1339aee700 1 --2- 172.21.15.38:0/1353156773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1334102930 0x7f1334111750 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1330011ef0 tx=0x7f1330009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:10.761 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.713+0000 7f132a7fc700 1 -- 172.21.15.38:0/1353156773 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1330010910 con 0x7f1334102930 2024-04-22T21:01:10.761 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.713+0000 7f132a7fc700 1 -- 172.21.15.38:0/1353156773 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1330010f00 con 0x7f1334102930 2024-04-22T21:01:10.761 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.713+0000 7f133bd52700 1 -- 172.21.15.38:0/1353156773 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1334111e90 con 0x7f1334102930 2024-04-22T21:01:10.761 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.713+0000 7f132a7fc700 1 -- 172.21.15.38:0/1353156773 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f133001ae30 con 0x7f1334102930 2024-04-22T21:01:10.761 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.713+0000 7f133bd52700 1 -- 172.21.15.38:0/1353156773 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1334112a00 con 0x7f1334102930 2024-04-22T21:01:10.761 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.714+0000 7f132a7fc700 1 -- 172.21.15.38:0/1353156773 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f133002d3f0 con 0x7f1334102930 2024-04-22T21:01:10.761 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.714+0000 7f133bd52700 1 -- 172.21.15.38:0/1353156773 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133405f5f0 con 0x7f1334102930 2024-04-22T21:01:10.761 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.714+0000 7f132a7fc700 1 --2- 172.21.15.38:0/1353156773 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f1320036740 0x7f1320038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:10.761 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.714+0000 7f132a7fc700 1 -- 172.21.15.38:0/1353156773 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f133004a4f0 con 0x7f1334102930 2024-04-22T21:01:10.761 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.714+0000 7f13392ed700 1 --2- 172.21.15.38:0/1353156773 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f1320036740 0x7f1320038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:10.761 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.715+0000 7f13392ed700 1 --2- 172.21.15.38:0/1353156773 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f1320036740 0x7f1320038bf0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1324009990 tx=0x7f1324006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:10.761 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.719+0000 7f132a7fc700 1 -- 172.21.15.38:0/1353156773 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1330018b00 con 0x7f1334102930 2024-04-22T21:01:10.761 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.954+0000 7f133bd52700 1 -- 172.21.15.38:0/1353156773 --> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f13340ff470 con 0x7f1320036740 2024-04-22T21:01:10.762 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.958+0000 7f132a7fc700 1 -- 172.21.15.38:0/1353156773 <== mgr.14120 v2:172.21.15.38:6800/2551561659 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f13340ff470 con 0x7f1320036740 2024-04-22T21:01:10.762 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.960+0000 7f133bd52700 1 -- 172.21.15.38:0/1353156773 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f1320036740 msgr2=0x7f1320038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:10.762 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.960+0000 7f133bd52700 1 --2- 172.21.15.38:0/1353156773 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f1320036740 0x7f1320038bf0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1324009990 tx=0x7f1324006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:10.762 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.960+0000 7f133bd52700 1 -- 172.21.15.38:0/1353156773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1334102930 msgr2=0x7f1334111750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:10.762 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.960+0000 7f133bd52700 1 --2- 172.21.15.38:0/1353156773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1334102930 0x7f1334111750 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1330011ef0 tx=0x7f1330009250 comp rx=0 tx=0).stop 2024-04-22T21:01:10.762 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.960+0000 7f133bd52700 1 -- 172.21.15.38:0/1353156773 shutdown_connections 2024-04-22T21:01:10.762 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.960+0000 7f133bd52700 1 --2- 172.21.15.38:0/1353156773 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f1320036740 0x7f1320038bf0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:10.762 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.960+0000 7f133bd52700 1 --2- 172.21.15.38:0/1353156773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1334102930 0x7f1334111750 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:10.762 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.960+0000 7f133bd52700 1 -- 172.21.15.38:0/1353156773 >> 172.21.15.38:0/1353156773 conn(0x7f13340fe140 msgr2=0x7f13340feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:10.762 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.961+0000 7f133bd52700 1 -- 172.21.15.38:0/1353156773 shutdown_connections 2024-04-22T21:01:10.762 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:09.961+0000 7f133bd52700 1 -- 172.21.15.38:0/1353156773 wait complete. 2024-04-22T21:01:10.762 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying mgr service with default placement... 2024-04-22T21:01:11.227 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:10 smithi038 ceph-mon[26374]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:01:11.228 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:10 smithi038 ceph-mon[26374]: Saving service mon spec with placement count:5 2024-04-22T21:01:11.228 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:10 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:13.079 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:13 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:13.080 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:13 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:13.080 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:13 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:13.087 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-22T21:01:13.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.240+0000 7f942e85f700 1 Processor -- start 2024-04-22T21:01:13.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.240+0000 7f942e85f700 1 -- start start 2024-04-22T21:01:13.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.240+0000 7f942e85f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9428102990 0x7f9428102da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:13.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.240+0000 7f942e85f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94281032e0 con 0x7f9428102990 2024-04-22T21:01:13.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.241+0000 7f9427fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9428102990 0x7f9428102da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:13.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.241+0000 7f9427fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9428102990 0x7f9428102da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39122/0 (socket says 172.21.15.38:39122) 2024-04-22T21:01:13.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.241+0000 7f9427fff700 1 -- 172.21.15.38:0/1751945960 learned_addr learned my addr 172.21.15.38:0/1751945960 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:13.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.241+0000 7f9427fff700 1 -- 172.21.15.38:0/1751945960 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9428103420 con 0x7f9428102990 2024-04-22T21:01:13.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.241+0000 7f9427fff700 1 --2- 172.21.15.38:0/1751945960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9428102990 0x7f9428102da0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f94180098e0 tx=0x7f9418009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daaaebeeac26f0e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:13.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.241+0000 7f9426ffd700 1 -- 172.21.15.38:0/1751945960 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f941800b8f0 con 0x7f9428102990 2024-04-22T21:01:13.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.241+0000 7f9426ffd700 1 -- 172.21.15.38:0/1751945960 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9418011420 con 0x7f9428102990 2024-04-22T21:01:13.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.241+0000 7f9426ffd700 1 -- 172.21.15.38:0/1751945960 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9418010540 con 0x7f9428102990 2024-04-22T21:01:13.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.242+0000 7f942e85f700 1 -- 172.21.15.38:0/1751945960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9428102990 msgr2=0x7f9428102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:13.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.242+0000 7f942e85f700 1 --2- 172.21.15.38:0/1751945960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9428102990 0x7f9428102da0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f94180098e0 tx=0x7f9418009250 comp rx=0 tx=0).stop 2024-04-22T21:01:13.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.242+0000 7f942e85f700 1 -- 172.21.15.38:0/1751945960 shutdown_connections 2024-04-22T21:01:13.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.242+0000 7f942e85f700 1 --2- 172.21.15.38:0/1751945960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9428102990 0x7f9428102da0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:13.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.242+0000 7f942e85f700 1 -- 172.21.15.38:0/1751945960 >> 172.21.15.38:0/1751945960 conn(0x7f94280fe110 msgr2=0x7f9428100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:13.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.242+0000 7f942e85f700 1 -- 172.21.15.38:0/1751945960 shutdown_connections 2024-04-22T21:01:13.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.242+0000 7f942e85f700 1 -- 172.21.15.38:0/1751945960 wait complete. 2024-04-22T21:01:13.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.243+0000 7f942e85f700 1 Processor -- start 2024-04-22T21:01:13.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.243+0000 7f942e85f700 1 -- start start 2024-04-22T21:01:13.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.244+0000 7f942e85f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9428102990 0x7f9428111710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:13.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.244+0000 7f942e85f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9428111c50 con 0x7f9428102990 2024-04-22T21:01:13.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.244+0000 7f9427fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9428102990 0x7f9428111710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:13.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.244+0000 7f9427fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9428102990 0x7f9428111710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39126/0 (socket says 172.21.15.38:39126) 2024-04-22T21:01:13.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.244+0000 7f9427fff700 1 -- 172.21.15.38:0/135025510 learned_addr learned my addr 172.21.15.38:0/135025510 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:13.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.244+0000 7f9427fff700 1 -- 172.21.15.38:0/135025510 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94180094f0 con 0x7f9428102990 2024-04-22T21:01:13.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.244+0000 7f9427fff700 1 --2- 172.21.15.38:0/135025510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9428102990 0x7f9428111710 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f94180090b0 tx=0x7f9418009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:13.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.244+0000 7f9424ff9700 1 -- 172.21.15.38:0/135025510 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94180107b0 con 0x7f9428102990 2024-04-22T21:01:13.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.245+0000 7f9424ff9700 1 -- 172.21.15.38:0/135025510 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9418010de0 con 0x7f9428102990 2024-04-22T21:01:13.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.245+0000 7f942e85f700 1 -- 172.21.15.38:0/135025510 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9428111e50 con 0x7f9428102990 2024-04-22T21:01:13.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.245+0000 7f9424ff9700 1 -- 172.21.15.38:0/135025510 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f941801ae50 con 0x7f9428102990 2024-04-22T21:01:13.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.245+0000 7f942e85f700 1 -- 172.21.15.38:0/135025510 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94281129c0 con 0x7f9428102990 2024-04-22T21:01:13.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.245+0000 7f9424ff9700 1 -- 172.21.15.38:0/135025510 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f941802d3f0 con 0x7f9428102990 2024-04-22T21:01:13.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.246+0000 7f9424ff9700 1 --2- 172.21.15.38:0/135025510 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f9410036810 0x7f9410038cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:13.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.246+0000 7f9424ff9700 1 -- 172.21.15.38:0/135025510 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f941804a430 con 0x7f9428102990 2024-04-22T21:01:13.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.246+0000 7f94277fe700 1 --2- 172.21.15.38:0/135025510 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f9410036810 0x7f9410038cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:13.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.246+0000 7f942e85f700 1 -- 172.21.15.38:0/135025510 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f942805f5f0 con 0x7f9428102990 2024-04-22T21:01:13.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.247+0000 7f94277fe700 1 --2- 172.21.15.38:0/135025510 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f9410036810 0x7f9410038cc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f941c009990 tx=0x7f941c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:13.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.251+0000 7f9424ff9700 1 -- 172.21.15.38:0/135025510 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f941802b8e0 con 0x7f9428102990 2024-04-22T21:01:13.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.459+0000 7f942e85f700 1 -- 172.21.15.38:0/135025510 --> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f94280ff3c0 con 0x7f9410036810 2024-04-22T21:01:13.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.464+0000 7f9424ff9700 1 -- 172.21.15.38:0/135025510 <== mgr.14120 v2:172.21.15.38:6800/2551561659 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f94280ff3c0 con 0x7f9410036810 2024-04-22T21:01:13.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.466+0000 7f942e85f700 1 -- 172.21.15.38:0/135025510 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f9410036810 msgr2=0x7f9410038cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:13.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.466+0000 7f942e85f700 1 --2- 172.21.15.38:0/135025510 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f9410036810 0x7f9410038cc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f941c009990 tx=0x7f941c006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:13.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.466+0000 7f942e85f700 1 -- 172.21.15.38:0/135025510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9428102990 msgr2=0x7f9428111710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:13.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.466+0000 7f942e85f700 1 --2- 172.21.15.38:0/135025510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9428102990 0x7f9428111710 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f94180090b0 tx=0x7f9418009250 comp rx=0 tx=0).stop 2024-04-22T21:01:13.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.466+0000 7f942e85f700 1 -- 172.21.15.38:0/135025510 shutdown_connections 2024-04-22T21:01:13.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.466+0000 7f942e85f700 1 --2- 172.21.15.38:0/135025510 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f9410036810 0x7f9410038cc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:13.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.466+0000 7f942e85f700 1 --2- 172.21.15.38:0/135025510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9428102990 0x7f9428111710 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:13.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.466+0000 7f942e85f700 1 -- 172.21.15.38:0/135025510 >> 172.21.15.38:0/135025510 conn(0x7f94280fe110 msgr2=0x7f94280fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:13.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.466+0000 7f942e85f700 1 -- 172.21.15.38:0/135025510 shutdown_connections 2024-04-22T21:01:13.092 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:12.467+0000 7f942e85f700 1 -- 172.21.15.38:0/135025510 wait complete. 2024-04-22T21:01:13.092 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying crash service with default placement... 2024-04-22T21:01:14.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:14 smithi038 ceph-mon[26374]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:01:14.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:14 smithi038 ceph-mon[26374]: Saving service mgr spec with placement count:2 2024-04-22T21:01:14.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:14 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:14.789 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-22T21:01:14.789 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.545+0000 7fe629b93700 1 Processor -- start 2024-04-22T21:01:14.789 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.545+0000 7fe629b93700 1 -- start start 2024-04-22T21:01:14.789 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.545+0000 7fe629b93700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe624078bb0 0x7fe624077240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:14.789 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.545+0000 7fe629b93700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe624078fc0 con 0x7fe624078bb0 2024-04-22T21:01:14.789 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.545+0000 7fe6237fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe624078bb0 0x7fe624077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:14.789 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.545+0000 7fe6237fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe624078bb0 0x7fe624077240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52130/0 (socket says 172.21.15.38:52130) 2024-04-22T21:01:14.789 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.545+0000 7fe6237fe700 1 -- 172.21.15.38:0/2785489601 learned_addr learned my addr 172.21.15.38:0/2785489601 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:14.789 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.546+0000 7fe6237fe700 1 -- 172.21.15.38:0/2785489601 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe624079100 con 0x7fe624078bb0 2024-04-22T21:01:14.789 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.546+0000 7fe6237fe700 1 --2- 172.21.15.38:0/2785489601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe624078bb0 0x7fe624077240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009e50 tx=0x7fe61c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27351952014ef338 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:14.789 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.546+0000 7fe6227fc700 1 -- 172.21.15.38:0/2785489601 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe61c0092e0 con 0x7fe624078bb0 2024-04-22T21:01:14.790 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.546+0000 7fe6227fc700 1 -- 172.21.15.38:0/2785489601 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe61c012430 con 0x7fe624078bb0 2024-04-22T21:01:14.790 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.546+0000 7fe6227fc700 1 -- 172.21.15.38:0/2785489601 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe61c0075c0 con 0x7fe624078bb0 2024-04-22T21:01:14.790 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.547+0000 7fe629b93700 1 -- 172.21.15.38:0/2785489601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe624078bb0 msgr2=0x7fe624077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:14.790 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.547+0000 7fe629b93700 1 --2- 172.21.15.38:0/2785489601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe624078bb0 0x7fe624077240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009e50 tx=0x7fe61c009840 comp rx=0 tx=0).stop 2024-04-22T21:01:14.790 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.547+0000 7fe629b93700 1 -- 172.21.15.38:0/2785489601 shutdown_connections 2024-04-22T21:01:14.790 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.547+0000 7fe629b93700 1 --2- 172.21.15.38:0/2785489601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe624078bb0 0x7fe624077240 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:14.790 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.547+0000 7fe629b93700 1 -- 172.21.15.38:0/2785489601 >> 172.21.15.38:0/2785489601 conn(0x7fe6240fe140 msgr2=0x7fe624100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:14.790 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.547+0000 7fe629b93700 1 -- 172.21.15.38:0/2785489601 shutdown_connections 2024-04-22T21:01:14.790 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.547+0000 7fe629b93700 1 -- 172.21.15.38:0/2785489601 wait complete. 2024-04-22T21:01:14.790 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.547+0000 7fe629b93700 1 Processor -- start 2024-04-22T21:01:14.790 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.547+0000 7fe629b93700 1 -- start start 2024-04-22T21:01:14.790 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.548+0000 7fe629b93700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe624078bb0 0x7fe6241138d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:14.790 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.548+0000 7fe629b93700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe624113e10 con 0x7fe624078bb0 2024-04-22T21:01:14.791 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.548+0000 7fe6237fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe624078bb0 0x7fe6241138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:14.791 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.548+0000 7fe6237fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe624078bb0 0x7fe6241138d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52136/0 (socket says 172.21.15.38:52136) 2024-04-22T21:01:14.791 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.548+0000 7fe6237fe700 1 -- 172.21.15.38:0/884974286 learned_addr learned my addr 172.21.15.38:0/884974286 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:14.791 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.548+0000 7fe6237fe700 1 -- 172.21.15.38:0/884974286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61c009ae0 con 0x7fe624078bb0 2024-04-22T21:01:14.791 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.548+0000 7fe6237fe700 1 --2- 172.21.15.38:0/884974286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe624078bb0 0x7fe6241138d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009fd0 tx=0x7fe61c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:14.791 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.548+0000 7fe61bfff700 1 -- 172.21.15.38:0/884974286 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe61c0092e0 con 0x7fe624078bb0 2024-04-22T21:01:14.791 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.548+0000 7fe61bfff700 1 -- 172.21.15.38:0/884974286 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe61c007d50 con 0x7fe624078bb0 2024-04-22T21:01:14.791 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.548+0000 7fe629b93700 1 -- 172.21.15.38:0/884974286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe624114010 con 0x7fe624078bb0 2024-04-22T21:01:14.791 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.548+0000 7fe61bfff700 1 -- 172.21.15.38:0/884974286 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe61c01ca90 con 0x7fe624078bb0 2024-04-22T21:01:14.791 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.548+0000 7fe629b93700 1 -- 172.21.15.38:0/884974286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe624114b80 con 0x7fe624078bb0 2024-04-22T21:01:14.791 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.549+0000 7fe61bfff700 1 -- 172.21.15.38:0/884974286 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fe61c019070 con 0x7fe624078bb0 2024-04-22T21:01:14.791 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.549+0000 7fe629b93700 1 -- 172.21.15.38:0/884974286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe62405f5f0 con 0x7fe624078bb0 2024-04-22T21:01:14.791 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.549+0000 7fe61bfff700 1 --2- 172.21.15.38:0/884974286 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fe610036790 0x7fe610038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:14.791 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.550+0000 7fe61bfff700 1 -- 172.21.15.38:0/884974286 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe61c04a140 con 0x7fe624078bb0 2024-04-22T21:01:14.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.550+0000 7fe622ffd700 1 --2- 172.21.15.38:0/884974286 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fe610036790 0x7fe610038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:14.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.550+0000 7fe622ffd700 1 --2- 172.21.15.38:0/884974286 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fe610036790 0x7fe610038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009990 tx=0x7fe60c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:14.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.554+0000 7fe61bfff700 1 -- 172.21.15.38:0/884974286 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe61c019910 con 0x7fe624078bb0 2024-04-22T21:01:14.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.767+0000 7fe629b93700 1 -- 172.21.15.38:0/884974286 --> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6240ff370 con 0x7fe610036790 2024-04-22T21:01:14.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.772+0000 7fe61bfff700 1 -- 172.21.15.38:0/884974286 <== mgr.14120 v2:172.21.15.38:6800/2551561659 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fe6240ff370 con 0x7fe610036790 2024-04-22T21:01:14.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.775+0000 7fe629b93700 1 -- 172.21.15.38:0/884974286 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fe610036790 msgr2=0x7fe610038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:14.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.775+0000 7fe629b93700 1 --2- 172.21.15.38:0/884974286 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fe610036790 0x7fe610038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009990 tx=0x7fe60c006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:14.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.775+0000 7fe629b93700 1 -- 172.21.15.38:0/884974286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe624078bb0 msgr2=0x7fe6241138d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:14.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.775+0000 7fe629b93700 1 --2- 172.21.15.38:0/884974286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe624078bb0 0x7fe6241138d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009fd0 tx=0x7fe61c009840 comp rx=0 tx=0).stop 2024-04-22T21:01:14.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.775+0000 7fe629b93700 1 -- 172.21.15.38:0/884974286 shutdown_connections 2024-04-22T21:01:14.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.775+0000 7fe629b93700 1 --2- 172.21.15.38:0/884974286 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fe610036790 0x7fe610038c40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:14.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.775+0000 7fe629b93700 1 --2- 172.21.15.38:0/884974286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe624078bb0 0x7fe6241138d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:14.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.775+0000 7fe629b93700 1 -- 172.21.15.38:0/884974286 >> 172.21.15.38:0/884974286 conn(0x7fe6240fe140 msgr2=0x7fe6240fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:14.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.775+0000 7fe629b93700 1 -- 172.21.15.38:0/884974286 shutdown_connections 2024-04-22T21:01:14.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:13.775+0000 7fe629b93700 1 -- 172.21.15.38:0/884974286 wait complete. 2024-04-22T21:01:14.793 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying prometheus service with default placement... 2024-04-22T21:01:15.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:15 smithi038 ceph-mon[26374]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:01:15.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:15 smithi038 ceph-mon[26374]: Saving service crash spec with placement * 2024-04-22T21:01:15.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:15 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:16.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-22T21:01:16.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.260+0000 7f9912673700 1 Processor -- start 2024-04-22T21:01:16.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.260+0000 7f9912673700 1 -- start start 2024-04-22T21:01:16.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.260+0000 7f9912673700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f990c1018b0 0x7f990c101cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:16.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.260+0000 7f9912673700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c102200 con 0x7f990c1018b0 2024-04-22T21:01:16.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.260+0000 7f990bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f990c1018b0 0x7f990c101cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:16.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.260+0000 7f990bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f990c1018b0 0x7f990c101cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52144/0 (socket says 172.21.15.38:52144) 2024-04-22T21:01:16.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.260+0000 7f990bfff700 1 -- 172.21.15.38:0/2398552911 learned_addr learned my addr 172.21.15.38:0/2398552911 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:16.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.260+0000 7f990bfff700 1 -- 172.21.15.38:0/2398552911 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c102340 con 0x7f990c1018b0 2024-04-22T21:01:16.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.261+0000 7f990bfff700 1 --2- 172.21.15.38:0/2398552911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f990c1018b0 0x7f990c101cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0098e0 tx=0x7f98fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6693bffd42554f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:16.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.261+0000 7f990affd700 1 -- 172.21.15.38:0/2398552911 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98fc00b8f0 con 0x7f990c1018b0 2024-04-22T21:01:16.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.261+0000 7f990affd700 1 -- 172.21.15.38:0/2398552911 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98fc011420 con 0x7f990c1018b0 2024-04-22T21:01:16.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.261+0000 7f990affd700 1 -- 172.21.15.38:0/2398552911 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98fc010540 con 0x7f990c1018b0 2024-04-22T21:01:16.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.262+0000 7f9912673700 1 -- 172.21.15.38:0/2398552911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f990c1018b0 msgr2=0x7f990c101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:16.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.262+0000 7f9912673700 1 --2- 172.21.15.38:0/2398552911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f990c1018b0 0x7f990c101cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0098e0 tx=0x7f98fc009250 comp rx=0 tx=0).stop 2024-04-22T21:01:16.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.262+0000 7f9912673700 1 -- 172.21.15.38:0/2398552911 shutdown_connections 2024-04-22T21:01:16.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.262+0000 7f9912673700 1 --2- 172.21.15.38:0/2398552911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f990c1018b0 0x7f990c101cc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:16.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.262+0000 7f9912673700 1 -- 172.21.15.38:0/2398552911 >> 172.21.15.38:0/2398552911 conn(0x7f990c0fd010 msgr2=0x7f990c0ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:16.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.262+0000 7f9912673700 1 -- 172.21.15.38:0/2398552911 shutdown_connections 2024-04-22T21:01:16.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.262+0000 7f9912673700 1 -- 172.21.15.38:0/2398552911 wait complete. 2024-04-22T21:01:16.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.262+0000 7f9912673700 1 Processor -- start 2024-04-22T21:01:16.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.262+0000 7f9912673700 1 -- start start 2024-04-22T21:01:16.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.263+0000 7f9912673700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f990c1018b0 0x7f990c076b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:16.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.263+0000 7f9912673700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c112270 con 0x7f990c1018b0 2024-04-22T21:01:16.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.263+0000 7f990bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f990c1018b0 0x7f990c076b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:16.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.263+0000 7f990bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f990c1018b0 0x7f990c076b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52148/0 (socket says 172.21.15.38:52148) 2024-04-22T21:01:16.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.263+0000 7f990bfff700 1 -- 172.21.15.38:0/1683433971 learned_addr learned my addr 172.21.15.38:0/1683433971 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:16.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.263+0000 7f990bfff700 1 -- 172.21.15.38:0/1683433971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc0094f0 con 0x7f990c1018b0 2024-04-22T21:01:16.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.263+0000 7f990bfff700 1 --2- 172.21.15.38:0/1683433971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f990c1018b0 0x7f990c076b30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f98fc006d40 tx=0x7f98fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:16.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.263+0000 7f9908ff9700 1 -- 172.21.15.38:0/1683433971 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98fc00beb0 con 0x7f990c1018b0 2024-04-22T21:01:16.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.263+0000 7f9912673700 1 -- 172.21.15.38:0/1683433971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f990c116880 con 0x7f990c1018b0 2024-04-22T21:01:16.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.263+0000 7f9908ff9700 1 -- 172.21.15.38:0/1683433971 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98fc010c80 con 0x7f990c1018b0 2024-04-22T21:01:16.017 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.264+0000 7f9908ff9700 1 -- 172.21.15.38:0/1683433971 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98fc024d40 con 0x7f990c1018b0 2024-04-22T21:01:16.017 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.264+0000 7f9912673700 1 -- 172.21.15.38:0/1683433971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990c116c50 con 0x7f990c1018b0 2024-04-22T21:01:16.017 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.264+0000 7f9908ff9700 1 -- 172.21.15.38:0/1683433971 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f98fc01a430 con 0x7f990c1018b0 2024-04-22T21:01:16.017 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.264+0000 7f9912673700 1 -- 172.21.15.38:0/1683433971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990c04bed0 con 0x7f990c1018b0 2024-04-22T21:01:16.017 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.265+0000 7f9908ff9700 1 --2- 172.21.15.38:0/1683433971 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f98f4036810 0x7f98f4038cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:16.017 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.265+0000 7f9908ff9700 1 -- 172.21.15.38:0/1683433971 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f98fc04a200 con 0x7f990c1018b0 2024-04-22T21:01:16.017 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.265+0000 7f990b7fe700 1 --2- 172.21.15.38:0/1683433971 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f98f4036810 0x7f98f4038cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:16.017 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.265+0000 7f990b7fe700 1 --2- 172.21.15.38:0/1683433971 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f98f4036810 0x7f98f4038cc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9900009990 tx=0x7f9900006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:16.017 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.270+0000 7f9908ff9700 1 -- 172.21.15.38:0/1683433971 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f98fc018940 con 0x7f990c1018b0 2024-04-22T21:01:16.017 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.486+0000 7f9912673700 1 -- 172.21.15.38:0/1683433971 --> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f990c0fe3e0 con 0x7f98f4036810 2024-04-22T21:01:16.017 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.491+0000 7f9908ff9700 1 -- 172.21.15.38:0/1683433971 <== mgr.14120 v2:172.21.15.38:6800/2551561659 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f990c0fe3e0 con 0x7f98f4036810 2024-04-22T21:01:16.017 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.493+0000 7f9912673700 1 -- 172.21.15.38:0/1683433971 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f98f4036810 msgr2=0x7f98f4038cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:16.017 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.493+0000 7f9912673700 1 --2- 172.21.15.38:0/1683433971 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f98f4036810 0x7f98f4038cc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9900009990 tx=0x7f9900006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:16.017 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.493+0000 7f9912673700 1 -- 172.21.15.38:0/1683433971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f990c1018b0 msgr2=0x7f990c076b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:16.018 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.493+0000 7f9912673700 1 --2- 172.21.15.38:0/1683433971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f990c1018b0 0x7f990c076b30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f98fc006d40 tx=0x7f98fc009250 comp rx=0 tx=0).stop 2024-04-22T21:01:16.018 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.494+0000 7f9912673700 1 -- 172.21.15.38:0/1683433971 shutdown_connections 2024-04-22T21:01:16.018 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.494+0000 7f9912673700 1 --2- 172.21.15.38:0/1683433971 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f98f4036810 0x7f98f4038cc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:16.018 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.494+0000 7f9912673700 1 --2- 172.21.15.38:0/1683433971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f990c1018b0 0x7f990c076b30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:16.018 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.494+0000 7f9912673700 1 -- 172.21.15.38:0/1683433971 >> 172.21.15.38:0/1683433971 conn(0x7f990c0fd010 msgr2=0x7f990c0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:16.018 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.494+0000 7f9912673700 1 -- 172.21.15.38:0/1683433971 shutdown_connections 2024-04-22T21:01:16.018 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:15.494+0000 7f9912673700 1 -- 172.21.15.38:0/1683433971 wait complete. 2024-04-22T21:01:16.018 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying grafana service with default placement... 2024-04-22T21:01:16.493 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:16 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:16.493 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:16 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:17.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:17 smithi038 ceph-mon[26374]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:01:17.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:17 smithi038 ceph-mon[26374]: Saving service prometheus spec with placement count:1 2024-04-22T21:01:17.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:17 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:17.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-22T21:01:17.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.610+0000 7fc3d4fe4700 1 Processor -- start 2024-04-22T21:01:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.610+0000 7fc3d4fe4700 1 -- start start 2024-04-22T21:01:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.610+0000 7fc3d4fe4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3d0100c30 0x7fc3d0101040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.610+0000 7fc3d4fe4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d0101580 con 0x7fc3d0100c30 2024-04-22T21:01:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.610+0000 7fc3ce59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3d0100c30 0x7fc3d0101040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.610+0000 7fc3ce59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3d0100c30 0x7fc3d0101040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52164/0 (socket says 172.21.15.38:52164) 2024-04-22T21:01:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.610+0000 7fc3ce59c700 1 -- 172.21.15.38:0/4257752894 learned_addr learned my addr 172.21.15.38:0/4257752894 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.611+0000 7fc3ce59c700 1 -- 172.21.15.38:0/4257752894 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d01016c0 con 0x7fc3d0100c30 2024-04-22T21:01:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.611+0000 7fc3ce59c700 1 --2- 172.21.15.38:0/4257752894 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3d0100c30 0x7fc3d0101040 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00098e0 tx=0x7fc3c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79859edb80ec02d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.611+0000 7fc3cd59a700 1 -- 172.21.15.38:0/4257752894 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3c000b860 con 0x7fc3d0100c30 2024-04-22T21:01:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.611+0000 7fc3cd59a700 1 -- 172.21.15.38:0/4257752894 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3c000be90 con 0x7fc3d0100c30 2024-04-22T21:01:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.611+0000 7fc3cd59a700 1 -- 172.21.15.38:0/4257752894 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3c00103c0 con 0x7fc3d0100c30 2024-04-22T21:01:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.612+0000 7fc3d4fe4700 1 -- 172.21.15.38:0/4257752894 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3d0100c30 msgr2=0x7fc3d0101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.612+0000 7fc3d4fe4700 1 --2- 172.21.15.38:0/4257752894 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3d0100c30 0x7fc3d0101040 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00098e0 tx=0x7fc3c0009250 comp rx=0 tx=0).stop 2024-04-22T21:01:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.612+0000 7fc3d4fe4700 1 -- 172.21.15.38:0/4257752894 shutdown_connections 2024-04-22T21:01:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.612+0000 7fc3d4fe4700 1 --2- 172.21.15.38:0/4257752894 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3d0100c30 0x7fc3d0101040 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.612+0000 7fc3d4fe4700 1 -- 172.21.15.38:0/4257752894 >> 172.21.15.38:0/4257752894 conn(0x7fc3d00fc730 msgr2=0x7fc3d00feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.612+0000 7fc3d4fe4700 1 -- 172.21.15.38:0/4257752894 shutdown_connections 2024-04-22T21:01:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.612+0000 7fc3d4fe4700 1 -- 172.21.15.38:0/4257752894 wait complete. 2024-04-22T21:01:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.613+0000 7fc3d4fe4700 1 Processor -- start 2024-04-22T21:01:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.613+0000 7fc3d4fe4700 1 -- start start 2024-04-22T21:01:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.613+0000 7fc3d4fe4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3d0100c30 0x7fc3d01116c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.613+0000 7fc3d4fe4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d0111c00 con 0x7fc3d0100c30 2024-04-22T21:01:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.613+0000 7fc3ce59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3d0100c30 0x7fc3d01116c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.613+0000 7fc3ce59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3d0100c30 0x7fc3d01116c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52180/0 (socket says 172.21.15.38:52180) 2024-04-22T21:01:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.613+0000 7fc3ce59c700 1 -- 172.21.15.38:0/4265002632 learned_addr learned my addr 172.21.15.38:0/4265002632 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.614+0000 7fc3ce59c700 1 -- 172.21.15.38:0/4265002632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c00094f0 con 0x7fc3d0100c30 2024-04-22T21:01:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.614+0000 7fc3ce59c700 1 --2- 172.21.15.38:0/4265002632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3d0100c30 0x7fc3d01116c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00090b0 tx=0x7fc3c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.614+0000 7fc3beffd700 1 -- 172.21.15.38:0/4265002632 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3c0010840 con 0x7fc3d0100c30 2024-04-22T21:01:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.614+0000 7fc3beffd700 1 -- 172.21.15.38:0/4265002632 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3c0010e70 con 0x7fc3d0100c30 2024-04-22T21:01:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.614+0000 7fc3d4fe4700 1 -- 172.21.15.38:0/4265002632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d0111e00 con 0x7fc3d0100c30 2024-04-22T21:01:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.614+0000 7fc3beffd700 1 -- 172.21.15.38:0/4265002632 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3c0024e40 con 0x7fc3d0100c30 2024-04-22T21:01:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.614+0000 7fc3d4fe4700 1 -- 172.21.15.38:0/4265002632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d0112970 con 0x7fc3d0100c30 2024-04-22T21:01:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.615+0000 7fc3beffd700 1 -- 172.21.15.38:0/4265002632 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fc3c001a4e0 con 0x7fc3d0100c30 2024-04-22T21:01:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.615+0000 7fc3d4fe4700 1 -- 172.21.15.38:0/4265002632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d005f5f0 con 0x7fc3d0100c30 2024-04-22T21:01:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.616+0000 7fc3beffd700 1 --2- 172.21.15.38:0/4265002632 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fc3b8036790 0x7fc3b8038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.616+0000 7fc3beffd700 1 -- 172.21.15.38:0/4265002632 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc3c004a750 con 0x7fc3d0100c30 2024-04-22T21:01:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.616+0000 7fc3cdd9b700 1 --2- 172.21.15.38:0/4265002632 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fc3b8036790 0x7fc3b8038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.616+0000 7fc3cdd9b700 1 --2- 172.21.15.38:0/4265002632 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fc3b8036790 0x7fc3b8038c40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009990 tx=0x7fc3c4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.621+0000 7fc3beffd700 1 -- 172.21.15.38:0/4265002632 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fc3c0018920 con 0x7fc3d0100c30 2024-04-22T21:01:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.833+0000 7fc3d4fe4700 1 -- 172.21.15.38:0/4265002632 --> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3d00fda20 con 0x7fc3b8036790 2024-04-22T21:01:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.839+0000 7fc3beffd700 1 -- 172.21.15.38:0/4265002632 <== mgr.14120 v2:172.21.15.38:6800/2551561659 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fc3d00fda20 con 0x7fc3b8036790 2024-04-22T21:01:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.841+0000 7fc3d4fe4700 1 -- 172.21.15.38:0/4265002632 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fc3b8036790 msgr2=0x7fc3b8038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.841+0000 7fc3d4fe4700 1 --2- 172.21.15.38:0/4265002632 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fc3b8036790 0x7fc3b8038c40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009990 tx=0x7fc3c4006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.841+0000 7fc3d4fe4700 1 -- 172.21.15.38:0/4265002632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3d0100c30 msgr2=0x7fc3d01116c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.841+0000 7fc3d4fe4700 1 --2- 172.21.15.38:0/4265002632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3d0100c30 0x7fc3d01116c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00090b0 tx=0x7fc3c0009250 comp rx=0 tx=0).stop 2024-04-22T21:01:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.841+0000 7fc3d4fe4700 1 -- 172.21.15.38:0/4265002632 shutdown_connections 2024-04-22T21:01:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.841+0000 7fc3d4fe4700 1 --2- 172.21.15.38:0/4265002632 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fc3b8036790 0x7fc3b8038c40 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.841+0000 7fc3d4fe4700 1 --2- 172.21.15.38:0/4265002632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3d0100c30 0x7fc3d01116c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.841+0000 7fc3d4fe4700 1 -- 172.21.15.38:0/4265002632 >> 172.21.15.38:0/4265002632 conn(0x7fc3d00fc730 msgr2=0x7fc3d00fd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.842+0000 7fc3d4fe4700 1 -- 172.21.15.38:0/4265002632 shutdown_connections 2024-04-22T21:01:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:16.842+0000 7fc3d4fe4700 1 -- 172.21.15.38:0/4265002632 wait complete. 2024-04-22T21:01:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying node-exporter service with default placement... 2024-04-22T21:01:18.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:18 smithi038 ceph-mon[26374]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:01:18.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:18 smithi038 ceph-mon[26374]: Saving service grafana spec with placement count:1 2024-04-22T21:01:19.211 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-22T21:01:19.211 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.244+0000 7f2e73eec700 1 Processor -- start 2024-04-22T21:01:19.211 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.244+0000 7f2e73eec700 1 -- start start 2024-04-22T21:01:19.211 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.244+0000 7f2e73eec700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100d30 0x7f2e6c101160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:19.211 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.244+0000 7f2e73eec700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c103b90 con 0x7f2e6c100d30 2024-04-22T21:01:19.212 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.244+0000 7f2e71c88700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100d30 0x7f2e6c101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:19.212 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.244+0000 7f2e71c88700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100d30 0x7f2e6c101160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52182/0 (socket says 172.21.15.38:52182) 2024-04-22T21:01:19.212 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.244+0000 7f2e71c88700 1 -- 172.21.15.38:0/1871745909 learned_addr learned my addr 172.21.15.38:0/1871745909 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:19.212 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.245+0000 7f2e71c88700 1 -- 172.21.15.38:0/1871745909 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6c1016a0 con 0x7f2e6c100d30 2024-04-22T21:01:19.212 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.245+0000 7f2e71c88700 1 --2- 172.21.15.38:0/1871745909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100d30 0x7f2e6c101160 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2e680098e0 tx=0x7f2e68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b292b68eb28abc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:19.212 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.245+0000 7f2e70c86700 1 -- 172.21.15.38:0/1871745909 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e6800b860 con 0x7f2e6c100d30 2024-04-22T21:01:19.212 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.245+0000 7f2e70c86700 1 -- 172.21.15.38:0/1871745909 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e6800be90 con 0x7f2e6c100d30 2024-04-22T21:01:19.212 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.245+0000 7f2e70c86700 1 -- 172.21.15.38:0/1871745909 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e680103c0 con 0x7f2e6c100d30 2024-04-22T21:01:19.212 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.245+0000 7f2e73eec700 1 -- 172.21.15.38:0/1871745909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100d30 msgr2=0x7f2e6c101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:19.212 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.245+0000 7f2e73eec700 1 --2- 172.21.15.38:0/1871745909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100d30 0x7f2e6c101160 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2e680098e0 tx=0x7f2e68009250 comp rx=0 tx=0).stop 2024-04-22T21:01:19.212 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.246+0000 7f2e73eec700 1 -- 172.21.15.38:0/1871745909 shutdown_connections 2024-04-22T21:01:19.212 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.246+0000 7f2e73eec700 1 --2- 172.21.15.38:0/1871745909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100d30 0x7f2e6c101160 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:19.212 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.246+0000 7f2e73eec700 1 -- 172.21.15.38:0/1871745909 >> 172.21.15.38:0/1871745909 conn(0x7f2e6c0fc770 msgr2=0x7f2e6c0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:19.212 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.246+0000 7f2e73eec700 1 -- 172.21.15.38:0/1871745909 shutdown_connections 2024-04-22T21:01:19.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.246+0000 7f2e73eec700 1 -- 172.21.15.38:0/1871745909 wait complete. 2024-04-22T21:01:19.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.246+0000 7f2e73eec700 1 Processor -- start 2024-04-22T21:01:19.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.246+0000 7f2e73eec700 1 -- start start 2024-04-22T21:01:19.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.246+0000 7f2e73eec700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100d30 0x7f2e6c113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:19.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.246+0000 7f2e73eec700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c113ed0 con 0x7f2e6c100d30 2024-04-22T21:01:19.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.247+0000 7f2e71c88700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100d30 0x7f2e6c113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:19.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.247+0000 7f2e71c88700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100d30 0x7f2e6c113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52190/0 (socket says 172.21.15.38:52190) 2024-04-22T21:01:19.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.247+0000 7f2e71c88700 1 -- 172.21.15.38:0/699052750 learned_addr learned my addr 172.21.15.38:0/699052750 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:19.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.247+0000 7f2e71c88700 1 -- 172.21.15.38:0/699052750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e680094f0 con 0x7f2e6c100d30 2024-04-22T21:01:19.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.247+0000 7f2e71c88700 1 --2- 172.21.15.38:0/699052750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100d30 0x7f2e6c113990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2e680094c0 tx=0x7f2e68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:19.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.247+0000 7f2e5e7fc700 1 -- 172.21.15.38:0/699052750 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e680118f0 con 0x7f2e6c100d30 2024-04-22T21:01:19.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.247+0000 7f2e5e7fc700 1 -- 172.21.15.38:0/699052750 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e68010d60 con 0x7f2e6c100d30 2024-04-22T21:01:19.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.247+0000 7f2e73eec700 1 -- 172.21.15.38:0/699052750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6c1140d0 con 0x7f2e6c100d30 2024-04-22T21:01:19.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.247+0000 7f2e5e7fc700 1 -- 172.21.15.38:0/699052750 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e6801a430 con 0x7f2e6c100d30 2024-04-22T21:01:19.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.247+0000 7f2e73eec700 1 -- 172.21.15.38:0/699052750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e6c114c40 con 0x7f2e6c100d30 2024-04-22T21:01:19.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.248+0000 7f2e5e7fc700 1 -- 172.21.15.38:0/699052750 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f2e6801a590 con 0x7f2e6c100d30 2024-04-22T21:01:19.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.248+0000 7f2e73eec700 1 -- 172.21.15.38:0/699052750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e6c05f5f0 con 0x7f2e6c100d30 2024-04-22T21:01:19.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.248+0000 7f2e5e7fc700 1 --2- 172.21.15.38:0/699052750 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f2e58036790 0x7f2e58038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:19.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.249+0000 7f2e5e7fc700 1 -- 172.21.15.38:0/699052750 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2e6804a650 con 0x7f2e6c100d30 2024-04-22T21:01:19.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.249+0000 7f2e71487700 1 --2- 172.21.15.38:0/699052750 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f2e58036790 0x7f2e58038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:19.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.249+0000 7f2e71487700 1 --2- 172.21.15.38:0/699052750 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f2e58036790 0x7f2e58038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2e60009990 tx=0x7f2e60006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:19.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.253+0000 7f2e5e7fc700 1 -- 172.21.15.38:0/699052750 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2e6801a840 con 0x7f2e6c100d30 2024-04-22T21:01:19.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.461+0000 7f2e73eec700 1 -- 172.21.15.38:0/699052750 --> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e6c0fdba0 con 0x7f2e58036790 2024-04-22T21:01:19.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.465+0000 7f2e5e7fc700 1 -- 172.21.15.38:0/699052750 <== mgr.14120 v2:172.21.15.38:6800/2551561659 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f2e6c0fdba0 con 0x7f2e58036790 2024-04-22T21:01:19.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.467+0000 7f2e73eec700 1 -- 172.21.15.38:0/699052750 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f2e58036790 msgr2=0x7f2e58038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:19.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.467+0000 7f2e73eec700 1 --2- 172.21.15.38:0/699052750 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f2e58036790 0x7f2e58038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2e60009990 tx=0x7f2e60006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:19.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.467+0000 7f2e73eec700 1 -- 172.21.15.38:0/699052750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100d30 msgr2=0x7f2e6c113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:19.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.467+0000 7f2e73eec700 1 --2- 172.21.15.38:0/699052750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100d30 0x7f2e6c113990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2e680094c0 tx=0x7f2e68009250 comp rx=0 tx=0).stop 2024-04-22T21:01:19.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.467+0000 7f2e73eec700 1 -- 172.21.15.38:0/699052750 shutdown_connections 2024-04-22T21:01:19.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.467+0000 7f2e73eec700 1 --2- 172.21.15.38:0/699052750 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f2e58036790 0x7f2e58038c40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:19.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.467+0000 7f2e73eec700 1 --2- 172.21.15.38:0/699052750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100d30 0x7f2e6c113990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:19.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.467+0000 7f2e73eec700 1 -- 172.21.15.38:0/699052750 >> 172.21.15.38:0/699052750 conn(0x7f2e6c0fc770 msgr2=0x7f2e6c0fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:19.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.467+0000 7f2e73eec700 1 -- 172.21.15.38:0/699052750 shutdown_connections 2024-04-22T21:01:19.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:18.468+0000 7f2e73eec700 1 -- 172.21.15.38:0/699052750 wait complete. 2024-04-22T21:01:19.215 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying alertmanager service with default placement... 2024-04-22T21:01:19.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:19 smithi038 ceph-mon[26374]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:01:19.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:19 smithi038 ceph-mon[26374]: Saving service node-exporter spec with placement * 2024-04-22T21:01:19.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:19 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:19.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:19 smithi038 ceph-mon[26374]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:01:20.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-22T21:01:20.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.677+0000 7fccd5e86700 1 Processor -- start 2024-04-22T21:01:20.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.677+0000 7fccd5e86700 1 -- start start 2024-04-22T21:01:20.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.677+0000 7fccd5e86700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd0101880 0x7fccd0101c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:20.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.677+0000 7fccd5e86700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd01021d0 con 0x7fccd0101880 2024-04-22T21:01:20.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.677+0000 7fcccf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd0101880 0x7fccd0101c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:20.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.677+0000 7fcccf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd0101880 0x7fccd0101c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52194/0 (socket says 172.21.15.38:52194) 2024-04-22T21:01:20.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.677+0000 7fcccf7fe700 1 -- 172.21.15.38:0/2054880845 learned_addr learned my addr 172.21.15.38:0/2054880845 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:20.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.678+0000 7fcccf7fe700 1 -- 172.21.15.38:0/2054880845 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd0102310 con 0x7fccd0101880 2024-04-22T21:01:20.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.678+0000 7fcccf7fe700 1 --2- 172.21.15.38:0/2054880845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd0101880 0x7fccd0101c90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fccc00098e0 tx=0x7fccc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac61a3f9cb3ba36 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:20.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.678+0000 7fccce7fc700 1 -- 172.21.15.38:0/2054880845 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccc000b8f0 con 0x7fccd0101880 2024-04-22T21:01:20.492 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.678+0000 7fccce7fc700 1 -- 172.21.15.38:0/2054880845 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fccc0011420 con 0x7fccd0101880 2024-04-22T21:01:20.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.678+0000 7fccce7fc700 1 -- 172.21.15.38:0/2054880845 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccc0010540 con 0x7fccd0101880 2024-04-22T21:01:20.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.679+0000 7fccd5e86700 1 -- 172.21.15.38:0/2054880845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd0101880 msgr2=0x7fccd0101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:20.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.679+0000 7fccd5e86700 1 --2- 172.21.15.38:0/2054880845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd0101880 0x7fccd0101c90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fccc00098e0 tx=0x7fccc0009250 comp rx=0 tx=0).stop 2024-04-22T21:01:20.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.679+0000 7fccd5e86700 1 -- 172.21.15.38:0/2054880845 shutdown_connections 2024-04-22T21:01:20.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.679+0000 7fccd5e86700 1 --2- 172.21.15.38:0/2054880845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd0101880 0x7fccd0101c90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:20.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.679+0000 7fccd5e86700 1 -- 172.21.15.38:0/2054880845 >> 172.21.15.38:0/2054880845 conn(0x7fccd00fd040 msgr2=0x7fccd00ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:20.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.679+0000 7fccd5e86700 1 -- 172.21.15.38:0/2054880845 shutdown_connections 2024-04-22T21:01:20.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.679+0000 7fccd5e86700 1 -- 172.21.15.38:0/2054880845 wait complete. 2024-04-22T21:01:20.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.679+0000 7fccd5e86700 1 Processor -- start 2024-04-22T21:01:20.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.680+0000 7fccd5e86700 1 -- start start 2024-04-22T21:01:20.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.680+0000 7fccd5e86700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd0101880 0x7fccd0076b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:20.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.680+0000 7fccd5e86700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd01143d0 con 0x7fccd0101880 2024-04-22T21:01:20.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.680+0000 7fcccf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd0101880 0x7fccd0076b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:20.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.680+0000 7fcccf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd0101880 0x7fccd0076b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52210/0 (socket says 172.21.15.38:52210) 2024-04-22T21:01:20.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.680+0000 7fcccf7fe700 1 -- 172.21.15.38:0/3012004544 learned_addr learned my addr 172.21.15.38:0/3012004544 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:20.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.680+0000 7fcccf7fe700 1 -- 172.21.15.38:0/3012004544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc00094f0 con 0x7fccd0101880 2024-04-22T21:01:20.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.680+0000 7fcccf7fe700 1 --2- 172.21.15.38:0/3012004544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd0101880 0x7fccd0076b30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fccc00090b0 tx=0x7fccc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:20.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.681+0000 7fccb7fff700 1 -- 172.21.15.38:0/3012004544 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccc00107b0 con 0x7fccd0101880 2024-04-22T21:01:20.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.681+0000 7fccb7fff700 1 -- 172.21.15.38:0/3012004544 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fccc0010de0 con 0x7fccd0101880 2024-04-22T21:01:20.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.681+0000 7fccd5e86700 1 -- 172.21.15.38:0/3012004544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd0114510 con 0x7fccd0101880 2024-04-22T21:01:20.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.681+0000 7fccb7fff700 1 -- 172.21.15.38:0/3012004544 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccc001ae50 con 0x7fccd0101880 2024-04-22T21:01:20.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.681+0000 7fccd5e86700 1 -- 172.21.15.38:0/3012004544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd01149a0 con 0x7fccd0101880 2024-04-22T21:01:20.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.681+0000 7fccb7fff700 1 -- 172.21.15.38:0/3012004544 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fccc002d3f0 con 0x7fccd0101880 2024-04-22T21:01:20.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.682+0000 7fccb7fff700 1 --2- 172.21.15.38:0/3012004544 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fccb8036810 0x7fccb8038cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:20.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.682+0000 7fccb7fff700 1 -- 172.21.15.38:0/3012004544 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fccc004a0f0 con 0x7fccd0101880 2024-04-22T21:01:20.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.682+0000 7fccd5e86700 1 -- 172.21.15.38:0/3012004544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd004bed0 con 0x7fccd0101880 2024-04-22T21:01:20.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.682+0000 7fccceffd700 1 --2- 172.21.15.38:0/3012004544 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fccb8036810 0x7fccb8038cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:20.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.682+0000 7fccceffd700 1 --2- 172.21.15.38:0/3012004544 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fccb8036810 0x7fccb8038cc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fccc4009990 tx=0x7fccc4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:20.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.687+0000 7fccb7fff700 1 -- 172.21.15.38:0/3012004544 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fccc002b8e0 con 0x7fccd0101880 2024-04-22T21:01:20.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.894+0000 7fccd5e86700 1 -- 172.21.15.38:0/3012004544 --> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fccd00fe2f0 con 0x7fccb8036810 2024-04-22T21:01:20.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.898+0000 7fccb7fff700 1 -- 172.21.15.38:0/3012004544 <== mgr.14120 v2:172.21.15.38:6800/2551561659 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fccd00fe2f0 con 0x7fccb8036810 2024-04-22T21:01:20.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.900+0000 7fccd5e86700 1 -- 172.21.15.38:0/3012004544 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fccb8036810 msgr2=0x7fccb8038cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:20.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.900+0000 7fccd5e86700 1 --2- 172.21.15.38:0/3012004544 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fccb8036810 0x7fccb8038cc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fccc4009990 tx=0x7fccc4006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:20.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.900+0000 7fccd5e86700 1 -- 172.21.15.38:0/3012004544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd0101880 msgr2=0x7fccd0076b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:20.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.900+0000 7fccd5e86700 1 --2- 172.21.15.38:0/3012004544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd0101880 0x7fccd0076b30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fccc00090b0 tx=0x7fccc0009250 comp rx=0 tx=0).stop 2024-04-22T21:01:20.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.900+0000 7fccd5e86700 1 -- 172.21.15.38:0/3012004544 shutdown_connections 2024-04-22T21:01:20.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.900+0000 7fccd5e86700 1 --2- 172.21.15.38:0/3012004544 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fccb8036810 0x7fccb8038cc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:20.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.900+0000 7fccd5e86700 1 --2- 172.21.15.38:0/3012004544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd0101880 0x7fccd0076b30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:20.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.900+0000 7fccd5e86700 1 -- 172.21.15.38:0/3012004544 >> 172.21.15.38:0/3012004544 conn(0x7fccd00fd040 msgr2=0x7fccd00fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:20.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.901+0000 7fccd5e86700 1 -- 172.21.15.38:0/3012004544 shutdown_connections 2024-04-22T21:01:20.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:19.901+0000 7fccd5e86700 1 -- 172.21.15.38:0/3012004544 wait complete. 2024-04-22T21:01:21.100 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:20 smithi038 ceph-mon[26374]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:01:21.101 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:20 smithi038 ceph-mon[26374]: Saving service alertmanager spec with placement count:1 2024-04-22T21:01:21.101 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:20 smithi038 ceph-mon[26374]: from='mgr.14120 172.21.15.38:0/1872612510' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:21.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.899+0000 7fe664c25700 1 Processor -- start 2024-04-22T21:01:21.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.899+0000 7fe664c25700 1 -- start start 2024-04-22T21:01:21.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.899+0000 7fe664c25700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe660076b40 0x7fe660105550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:21.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.899+0000 7fe664c25700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe660076f50 con 0x7fe660076b40 2024-04-22T21:01:21.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.900+0000 7fe65e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe660076b40 0x7fe660105550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:21.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.900+0000 7fe65e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe660076b40 0x7fe660105550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52216/0 (socket says 172.21.15.38:52216) 2024-04-22T21:01:21.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.900+0000 7fe65e59c700 1 -- 172.21.15.38:0/3624229985 learned_addr learned my addr 172.21.15.38:0/3624229985 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:21.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.900+0000 7fe65e59c700 1 -- 172.21.15.38:0/3624229985 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe660105a90 con 0x7fe660076b40 2024-04-22T21:01:21.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.900+0000 7fe65e59c700 1 --2- 172.21.15.38:0/3624229985 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe660076b40 0x7fe660105550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe6500098e0 tx=0x7fe650009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581ede74c4c38dcf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:21.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.900+0000 7fe65d59a700 1 -- 172.21.15.38:0/3624229985 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe65000b8f0 con 0x7fe660076b40 2024-04-22T21:01:21.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.901+0000 7fe65d59a700 1 -- 172.21.15.38:0/3624229985 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe650011420 con 0x7fe660076b40 2024-04-22T21:01:21.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.901+0000 7fe65d59a700 1 -- 172.21.15.38:0/3624229985 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe650010540 con 0x7fe660076b40 2024-04-22T21:01:21.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.901+0000 7fe664c25700 1 -- 172.21.15.38:0/3624229985 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe660076b40 msgr2=0x7fe660105550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:21.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.901+0000 7fe664c25700 1 --2- 172.21.15.38:0/3624229985 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe660076b40 0x7fe660105550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe6500098e0 tx=0x7fe650009250 comp rx=0 tx=0).stop 2024-04-22T21:01:21.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.901+0000 7fe664c25700 1 -- 172.21.15.38:0/3624229985 shutdown_connections 2024-04-22T21:01:21.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.901+0000 7fe664c25700 1 --2- 172.21.15.38:0/3624229985 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe660076b40 0x7fe660105550 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:21.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.901+0000 7fe664c25700 1 -- 172.21.15.38:0/3624229985 >> 172.21.15.38:0/3624229985 conn(0x7fe6600fd040 msgr2=0x7fe6600ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:21.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.901+0000 7fe664c25700 1 -- 172.21.15.38:0/3624229985 shutdown_connections 2024-04-22T21:01:21.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.901+0000 7fe664c25700 1 -- 172.21.15.38:0/3624229985 wait complete. 2024-04-22T21:01:21.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.902+0000 7fe664c25700 1 Processor -- start 2024-04-22T21:01:21.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.902+0000 7fe664c25700 1 -- start start 2024-04-22T21:01:21.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.902+0000 7fe664c25700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe660076b40 0x7fe660102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:21.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.902+0000 7fe664c25700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6601033e0 con 0x7fe660076b40 2024-04-22T21:01:21.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.902+0000 7fe65e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe660076b40 0x7fe660102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:21.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.902+0000 7fe65e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe660076b40 0x7fe660102ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52230/0 (socket says 172.21.15.38:52230) 2024-04-22T21:01:21.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.902+0000 7fe65e59c700 1 -- 172.21.15.38:0/1554195213 learned_addr learned my addr 172.21.15.38:0/1554195213 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:21.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.902+0000 7fe65e59c700 1 -- 172.21.15.38:0/1554195213 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6500094f0 con 0x7fe660076b40 2024-04-22T21:01:21.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.903+0000 7fe65e59c700 1 --2- 172.21.15.38:0/1554195213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe660076b40 0x7fe660102ea0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe650011cf0 tx=0x7fe650009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:21.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.903+0000 7fe64effd700 1 -- 172.21.15.38:0/1554195213 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe650010800 con 0x7fe660076b40 2024-04-22T21:01:21.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.903+0000 7fe64effd700 1 -- 172.21.15.38:0/1554195213 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe650010e30 con 0x7fe660076b40 2024-04-22T21:01:21.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.903+0000 7fe664c25700 1 -- 172.21.15.38:0/1554195213 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe660101450 con 0x7fe660076b40 2024-04-22T21:01:21.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.903+0000 7fe64effd700 1 -- 172.21.15.38:0/1554195213 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe65001ae80 con 0x7fe660076b40 2024-04-22T21:01:21.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.903+0000 7fe664c25700 1 -- 172.21.15.38:0/1554195213 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe660101940 con 0x7fe660076b40 2024-04-22T21:01:21.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.904+0000 7fe64effd700 1 -- 172.21.15.38:0/1554195213 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fe65002d3f0 con 0x7fe660076b40 2024-04-22T21:01:21.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.904+0000 7fe664c25700 1 -- 172.21.15.38:0/1554195213 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66004bed0 con 0x7fe660076b40 2024-04-22T21:01:21.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.904+0000 7fe64effd700 1 --2- 172.21.15.38:0/1554195213 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fe6480367e0 0x7fe648038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:21.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.904+0000 7fe64effd700 1 -- 172.21.15.38:0/1554195213 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe65004a3a0 con 0x7fe660076b40 2024-04-22T21:01:21.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.904+0000 7fe65dd9b700 1 --2- 172.21.15.38:0/1554195213 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fe6480367e0 0x7fe648038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:21.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.904+0000 7fe65dd9b700 1 --2- 172.21.15.38:0/1554195213 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fe6480367e0 0x7fe648038c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe654009990 tx=0x7fe654006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:21.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:20.909+0000 7fe64effd700 1 -- 172.21.15.38:0/1554195213 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe65002b8e0 con 0x7fe660076b40 2024-04-22T21:01:21.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:21.105+0000 7fe664c25700 1 -- 172.21.15.38:0/1554195213 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fe660106880 con 0x7fe660076b40 2024-04-22T21:01:21.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:21.108+0000 7fe64effd700 1 -- 172.21.15.38:0/1554195213 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fe65002b2f0 con 0x7fe660076b40 2024-04-22T21:01:21.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:21.111+0000 7fe664c25700 1 -- 172.21.15.38:0/1554195213 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fe6480367e0 msgr2=0x7fe648038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:21.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:21.111+0000 7fe664c25700 1 --2- 172.21.15.38:0/1554195213 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fe6480367e0 0x7fe648038c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe654009990 tx=0x7fe654006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:21.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:21.111+0000 7fe664c25700 1 -- 172.21.15.38:0/1554195213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe660076b40 msgr2=0x7fe660102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:21.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:21.111+0000 7fe664c25700 1 --2- 172.21.15.38:0/1554195213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe660076b40 0x7fe660102ea0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe650011cf0 tx=0x7fe650009250 comp rx=0 tx=0).stop 2024-04-22T21:01:21.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:21.111+0000 7fe664c25700 1 -- 172.21.15.38:0/1554195213 shutdown_connections 2024-04-22T21:01:21.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:21.111+0000 7fe664c25700 1 --2- 172.21.15.38:0/1554195213 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fe6480367e0 0x7fe648038c90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:21.619 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:21.111+0000 7fe664c25700 1 --2- 172.21.15.38:0/1554195213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe660076b40 0x7fe660102ea0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:21.619 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:21.111+0000 7fe664c25700 1 -- 172.21.15.38:0/1554195213 >> 172.21.15.38:0/1554195213 conn(0x7fe6600fd040 msgr2=0x7fe6600fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:21.619 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:21.111+0000 7fe664c25700 1 -- 172.21.15.38:0/1554195213 shutdown_connections 2024-04-22T21:01:21.619 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:21.111+0000 7fe664c25700 1 -- 172.21.15.38:0/1554195213 wait complete. 2024-04-22T21:01:22.283 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:22 smithi038 ceph-mon[26374]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:01:22.284 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:22 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1554195213' entity='client.admin' 2024-04-22T21:01:22.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.069+0000 7f490e7ba700 1 Processor -- start 2024-04-22T21:01:22.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.069+0000 7f490e7ba700 1 -- start start 2024-04-22T21:01:22.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.069+0000 7f490e7ba700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4908102cf0 0x7f4908103100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:22.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.069+0000 7f490e7ba700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49081036d0 con 0x7f4908102cf0 2024-04-22T21:01:22.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.070+0000 7f4907fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4908102cf0 0x7f4908103100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:22.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.070+0000 7f4907fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4908102cf0 0x7f4908103100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52246/0 (socket says 172.21.15.38:52246) 2024-04-22T21:01:22.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.070+0000 7f4907fff700 1 -- 172.21.15.38:0/2873371547 learned_addr learned my addr 172.21.15.38:0/2873371547 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:22.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.070+0000 7f4907fff700 1 -- 172.21.15.38:0/2873371547 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4908103810 con 0x7f4908102cf0 2024-04-22T21:01:22.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.070+0000 7f4907fff700 1 --2- 172.21.15.38:0/2873371547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4908102cf0 0x7f4908103100 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f48f80098e0 tx=0x7f48f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599cd592a8a15803 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:22.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.070+0000 7f4906ffd700 1 -- 172.21.15.38:0/2873371547 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48f800b860 con 0x7f4908102cf0 2024-04-22T21:01:22.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.070+0000 7f4906ffd700 1 -- 172.21.15.38:0/2873371547 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48f800be90 con 0x7f4908102cf0 2024-04-22T21:01:22.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.070+0000 7f4906ffd700 1 -- 172.21.15.38:0/2873371547 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48f80103c0 con 0x7f4908102cf0 2024-04-22T21:01:22.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.071+0000 7f490e7ba700 1 -- 172.21.15.38:0/2873371547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4908102cf0 msgr2=0x7f4908103100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:22.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.071+0000 7f490e7ba700 1 --2- 172.21.15.38:0/2873371547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4908102cf0 0x7f4908103100 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f48f80098e0 tx=0x7f48f8009250 comp rx=0 tx=0).stop 2024-04-22T21:01:22.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.071+0000 7f490e7ba700 1 -- 172.21.15.38:0/2873371547 shutdown_connections 2024-04-22T21:01:22.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.071+0000 7f490e7ba700 1 --2- 172.21.15.38:0/2873371547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4908102cf0 0x7f4908103100 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:22.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.071+0000 7f490e7ba700 1 -- 172.21.15.38:0/2873371547 >> 172.21.15.38:0/2873371547 conn(0x7f49080fc770 msgr2=0x7f49080feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:22.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.071+0000 7f490e7ba700 1 -- 172.21.15.38:0/2873371547 shutdown_connections 2024-04-22T21:01:22.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.071+0000 7f490e7ba700 1 -- 172.21.15.38:0/2873371547 wait complete. 2024-04-22T21:01:22.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.071+0000 7f490e7ba700 1 Processor -- start 2024-04-22T21:01:22.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.072+0000 7f490e7ba700 1 -- start start 2024-04-22T21:01:22.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.072+0000 7f490e7ba700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4908102cf0 0x7f4908113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:22.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.072+0000 7f490e7ba700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4908113ec0 con 0x7f4908102cf0 2024-04-22T21:01:22.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.072+0000 7f4907fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4908102cf0 0x7f4908113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:22.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.072+0000 7f4907fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4908102cf0 0x7f4908113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52254/0 (socket says 172.21.15.38:52254) 2024-04-22T21:01:22.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.072+0000 7f4907fff700 1 -- 172.21.15.38:0/2770597708 learned_addr learned my addr 172.21.15.38:0/2770597708 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:22.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.072+0000 7f4907fff700 1 -- 172.21.15.38:0/2770597708 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f80094f0 con 0x7f4908102cf0 2024-04-22T21:01:22.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.072+0000 7f4907fff700 1 --2- 172.21.15.38:0/2770597708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4908102cf0 0x7f4908113980 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f48f80094c0 tx=0x7f48f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:22.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.073+0000 7f4904ff9700 1 -- 172.21.15.38:0/2770597708 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48f80118f0 con 0x7f4908102cf0 2024-04-22T21:01:22.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.073+0000 7f4904ff9700 1 -- 172.21.15.38:0/2770597708 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48f8010d60 con 0x7f4908102cf0 2024-04-22T21:01:22.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.073+0000 7f490e7ba700 1 -- 172.21.15.38:0/2770597708 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49081140c0 con 0x7f4908102cf0 2024-04-22T21:01:22.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.073+0000 7f4904ff9700 1 -- 172.21.15.38:0/2770597708 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48f801a430 con 0x7f4908102cf0 2024-04-22T21:01:22.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.073+0000 7f490e7ba700 1 -- 172.21.15.38:0/2770597708 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4908114c30 con 0x7f4908102cf0 2024-04-22T21:01:22.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.073+0000 7f4904ff9700 1 -- 172.21.15.38:0/2770597708 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f48f801a590 con 0x7f4908102cf0 2024-04-22T21:01:22.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.074+0000 7f4904ff9700 1 --2- 172.21.15.38:0/2770597708 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f48f00367e0 0x7f48f0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:22.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.074+0000 7f4904ff9700 1 -- 172.21.15.38:0/2770597708 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f48f804a400 con 0x7f4908102cf0 2024-04-22T21:01:22.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.074+0000 7f490e7ba700 1 -- 172.21.15.38:0/2770597708 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f490805f5f0 con 0x7f4908102cf0 2024-04-22T21:01:22.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.074+0000 7f49077fe700 1 --2- 172.21.15.38:0/2770597708 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f48f00367e0 0x7f48f0038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:22.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.074+0000 7f49077fe700 1 --2- 172.21.15.38:0/2770597708 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f48f00367e0 0x7f48f0038c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f48fc009990 tx=0x7f48fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:22.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.079+0000 7f4904ff9700 1 -- 172.21.15.38:0/2770597708 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f48f802d3f0 con 0x7f4908102cf0 2024-04-22T21:01:22.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.288+0000 7f490e7ba700 1 -- 172.21.15.38:0/2770597708 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f490804bed0 con 0x7f4908102cf0 2024-04-22T21:01:22.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.292+0000 7f4904ff9700 1 -- 172.21.15.38:0/2770597708 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f48f8020e90 con 0x7f4908102cf0 2024-04-22T21:01:22.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.294+0000 7f490e7ba700 1 -- 172.21.15.38:0/2770597708 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f48f00367e0 msgr2=0x7f48f0038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:22.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.294+0000 7f490e7ba700 1 --2- 172.21.15.38:0/2770597708 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f48f00367e0 0x7f48f0038c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f48fc009990 tx=0x7f48fc006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:22.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.294+0000 7f490e7ba700 1 -- 172.21.15.38:0/2770597708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4908102cf0 msgr2=0x7f4908113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:22.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.294+0000 7f490e7ba700 1 --2- 172.21.15.38:0/2770597708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4908102cf0 0x7f4908113980 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f48f80094c0 tx=0x7f48f8009250 comp rx=0 tx=0).stop 2024-04-22T21:01:22.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.294+0000 7f490e7ba700 1 -- 172.21.15.38:0/2770597708 shutdown_connections 2024-04-22T21:01:22.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.294+0000 7f490e7ba700 1 --2- 172.21.15.38:0/2770597708 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f48f00367e0 0x7f48f0038c90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:22.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.294+0000 7f490e7ba700 1 --2- 172.21.15.38:0/2770597708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4908102cf0 0x7f4908113980 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:22.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.294+0000 7f490e7ba700 1 -- 172.21.15.38:0/2770597708 >> 172.21.15.38:0/2770597708 conn(0x7f49080fc770 msgr2=0x7f49080fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:22.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.294+0000 7f490e7ba700 1 -- 172.21.15.38:0/2770597708 shutdown_connections 2024-04-22T21:01:22.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:22.294+0000 7f490e7ba700 1 -- 172.21.15.38:0/2770597708 wait complete. 2024-04-22T21:01:22.877 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling the dashboard module... 2024-04-22T21:01:23.553 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:23 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/2770597708' entity='client.admin' 2024-04-22T21:01:24.612 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:24 smithi038 ceph-mon[26374]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:01:24.612 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:24 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3411673372' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-22T21:01:24.895 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.295+0000 7f6a0aefb700 1 Processor -- start 2024-04-22T21:01:24.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.295+0000 7f6a0aefb700 1 -- start start 2024-04-22T21:01:24.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.295+0000 7f6a0aefb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a04103f30 0x7f6a04104340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:24.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.295+0000 7f6a0aefb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a04070e20 con 0x7f6a04103f30 2024-04-22T21:01:24.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.295+0000 7f6a08c97700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a04103f30 0x7f6a04104340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:24.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.295+0000 7f6a08c97700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a04103f30 0x7f6a04104340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42302/0 (socket says 172.21.15.38:42302) 2024-04-22T21:01:24.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.295+0000 7f6a08c97700 1 -- 172.21.15.38:0/1399240553 learned_addr learned my addr 172.21.15.38:0/1399240553 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:24.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.295+0000 7f6a08c97700 1 -- 172.21.15.38:0/1399240553 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a04104880 con 0x7f6a04103f30 2024-04-22T21:01:24.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.295+0000 7f6a08c97700 1 --2- 172.21.15.38:0/1399240553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a04103f30 0x7f6a04104340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f69f80098e0 tx=0x7f69f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d517ca64c58f402 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:24.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.297+0000 7f6a037fe700 1 -- 172.21.15.38:0/1399240553 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69f800b860 con 0x7f6a04103f30 2024-04-22T21:01:24.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.297+0000 7f6a037fe700 1 -- 172.21.15.38:0/1399240553 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69f800be90 con 0x7f6a04103f30 2024-04-22T21:01:24.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.297+0000 7f6a037fe700 1 -- 172.21.15.38:0/1399240553 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69f80103c0 con 0x7f6a04103f30 2024-04-22T21:01:24.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.298+0000 7f6a0aefb700 1 -- 172.21.15.38:0/1399240553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a04103f30 msgr2=0x7f6a04104340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:24.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.298+0000 7f6a0aefb700 1 --2- 172.21.15.38:0/1399240553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a04103f30 0x7f6a04104340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f69f80098e0 tx=0x7f69f8009250 comp rx=0 tx=0).stop 2024-04-22T21:01:24.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.298+0000 7f6a0aefb700 1 -- 172.21.15.38:0/1399240553 shutdown_connections 2024-04-22T21:01:24.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.298+0000 7f6a0aefb700 1 --2- 172.21.15.38:0/1399240553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a04103f30 0x7f6a04104340 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:24.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.298+0000 7f6a0aefb700 1 -- 172.21.15.38:0/1399240553 >> 172.21.15.38:0/1399240553 conn(0x7f6a040fc770 msgr2=0x7f6a040feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:24.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.298+0000 7f6a0aefb700 1 -- 172.21.15.38:0/1399240553 shutdown_connections 2024-04-22T21:01:24.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.298+0000 7f6a0aefb700 1 -- 172.21.15.38:0/1399240553 wait complete. 2024-04-22T21:01:24.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.299+0000 7f6a0aefb700 1 Processor -- start 2024-04-22T21:01:24.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.299+0000 7f6a0aefb700 1 -- start start 2024-04-22T21:01:24.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.299+0000 7f6a0aefb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a04103f30 0x7f6a04111780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:24.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.299+0000 7f6a0aefb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a04111cc0 con 0x7f6a04103f30 2024-04-22T21:01:24.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.299+0000 7f6a08c97700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a04103f30 0x7f6a04111780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:24.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.299+0000 7f6a08c97700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a04103f30 0x7f6a04111780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42318/0 (socket says 172.21.15.38:42318) 2024-04-22T21:01:24.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.299+0000 7f6a08c97700 1 -- 172.21.15.38:0/3411673372 learned_addr learned my addr 172.21.15.38:0/3411673372 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:24.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.300+0000 7f6a08c97700 1 -- 172.21.15.38:0/3411673372 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f80094f0 con 0x7f6a04103f30 2024-04-22T21:01:24.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.300+0000 7f6a08c97700 1 --2- 172.21.15.38:0/3411673372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a04103f30 0x7f6a04111780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f69f8011c10 tx=0x7f69f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:24.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.300+0000 7f6a017fa700 1 -- 172.21.15.38:0/3411673372 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69f8010890 con 0x7f6a04103f30 2024-04-22T21:01:24.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.300+0000 7f6a017fa700 1 -- 172.21.15.38:0/3411673372 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69f8010ec0 con 0x7f6a04103f30 2024-04-22T21:01:24.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.300+0000 7f6a017fa700 1 -- 172.21.15.38:0/3411673372 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69f80243f0 con 0x7f6a04103f30 2024-04-22T21:01:24.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.300+0000 7f6a0aefb700 1 -- 172.21.15.38:0/3411673372 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a04111ec0 con 0x7f6a04103f30 2024-04-22T21:01:24.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.300+0000 7f6a0aefb700 1 -- 172.21.15.38:0/3411673372 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a04112a30 con 0x7f6a04103f30 2024-04-22T21:01:24.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.301+0000 7f6a017fa700 1 -- 172.21.15.38:0/3411673372 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f69f802d3f0 con 0x7f6a04103f30 2024-04-22T21:01:24.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.301+0000 7f6a017fa700 1 --2- 172.21.15.38:0/3411673372 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f69f40367e0 0x7f69f4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:24.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.301+0000 7f6a017fa700 1 -- 172.21.15.38:0/3411673372 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f69f804a210 con 0x7f6a04103f30 2024-04-22T21:01:24.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.301+0000 7f6a0aefb700 1 -- 172.21.15.38:0/3411673372 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a0405f5f0 con 0x7f6a04103f30 2024-04-22T21:01:24.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.301+0000 7f6a03fff700 1 --2- 172.21.15.38:0/3411673372 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f69f40367e0 0x7f69f4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:24.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.302+0000 7f6a03fff700 1 --2- 172.21.15.38:0/3411673372 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f69f40367e0 0x7f69f4038c90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f69f0009990 tx=0x7f69f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:24.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.306+0000 7f6a017fa700 1 -- 172.21.15.38:0/3411673372 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f69f802b8e0 con 0x7f6a04103f30 2024-04-22T21:01:24.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:23.558+0000 7f6a0aefb700 1 -- 172.21.15.38:0/3411673372 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f6a0404bed0 con 0x7f6a04103f30 2024-04-22T21:01:24.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:24.297+0000 7f6a017fa700 1 -- 172.21.15.38:0/3411673372 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 9) v1 ==== 43389+0+0 (secure 0 0 0) 0x7f69f80193c0 con 0x7f6a04103f30 2024-04-22T21:01:24.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:24.299+0000 7f6a017fa700 1 -- 172.21.15.38:0/3411673372 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f69f8017490 con 0x7f6a04103f30 2024-04-22T21:01:24.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:24.302+0000 7f6a0aefb700 1 -- 172.21.15.38:0/3411673372 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f69f40367e0 msgr2=0x7f69f4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:24.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:24.302+0000 7f6a0aefb700 1 --2- 172.21.15.38:0/3411673372 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f69f40367e0 0x7f69f4038c90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f69f0009990 tx=0x7f69f0006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:24.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:24.302+0000 7f6a0aefb700 1 -- 172.21.15.38:0/3411673372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a04103f30 msgr2=0x7f6a04111780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:24.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:24.302+0000 7f6a0aefb700 1 --2- 172.21.15.38:0/3411673372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a04103f30 0x7f6a04111780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f69f8011c10 tx=0x7f69f8009250 comp rx=0 tx=0).stop 2024-04-22T21:01:24.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:24.302+0000 7f6a0aefb700 1 -- 172.21.15.38:0/3411673372 shutdown_connections 2024-04-22T21:01:24.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:24.302+0000 7f6a0aefb700 1 --2- 172.21.15.38:0/3411673372 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7f69f40367e0 0x7f69f4038c90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:24.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:24.302+0000 7f6a0aefb700 1 --2- 172.21.15.38:0/3411673372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a04103f30 0x7f6a04111780 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:24.900 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:24.302+0000 7f6a0aefb700 1 -- 172.21.15.38:0/3411673372 >> 172.21.15.38:0/3411673372 conn(0x7f6a040fc770 msgr2=0x7f6a040fd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:24.900 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:24.302+0000 7f6a0aefb700 1 -- 172.21.15.38:0/3411673372 shutdown_connections 2024-04-22T21:01:24.900 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:24.302+0000 7f6a0aefb700 1 -- 172.21.15.38:0/3411673372 wait complete. 2024-04-22T21:01:25.398 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:25 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3411673372' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-22T21:01:25.399 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:25 smithi038 ceph-mon[26374]: mgrmap e9: smithi038.lxfgut(active, since 25s) 2024-04-22T21:01:26.369 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:26 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/4110024529' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T21:01:26.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-22T21:01:26.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-22T21:01:26.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T21:01:26.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "active_name": "smithi038.lxfgut", 2024-04-22T21:01:26.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T21:01:26.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-22T21:01:26.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.421+0000 7fb117aaa700 1 Processor -- start 2024-04-22T21:01:26.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.421+0000 7fb117aaa700 1 -- start start 2024-04-22T21:01:26.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.421+0000 7fb117aaa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb110071280 0x7fb110107970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:26.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.421+0000 7fb117aaa700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb110071690 con 0x7fb110071280 2024-04-22T21:01:26.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.422+0000 7fb115846700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb110071280 0x7fb110107970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:26.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.422+0000 7fb115846700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb110071280 0x7fb110107970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42342/0 (socket says 172.21.15.38:42342) 2024-04-22T21:01:26.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.422+0000 7fb115846700 1 -- 172.21.15.38:0/1258864340 learned_addr learned my addr 172.21.15.38:0/1258864340 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:26.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.422+0000 7fb115846700 1 -- 172.21.15.38:0/1258864340 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb110103360 con 0x7fb110071280 2024-04-22T21:01:26.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.422+0000 7fb115846700 1 --2- 172.21.15.38:0/1258864340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb110071280 0x7fb110107970 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0098e0 tx=0x7fb10c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c2e96665c9535e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:26.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.422+0000 7fb114844700 1 -- 172.21.15.38:0/1258864340 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb10c00b860 con 0x7fb110071280 2024-04-22T21:01:26.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.422+0000 7fb114844700 1 -- 172.21.15.38:0/1258864340 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb10c00be90 con 0x7fb110071280 2024-04-22T21:01:26.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.422+0000 7fb114844700 1 -- 172.21.15.38:0/1258864340 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb10c0103c0 con 0x7fb110071280 2024-04-22T21:01:26.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.423+0000 7fb117aaa700 1 -- 172.21.15.38:0/1258864340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb110071280 msgr2=0x7fb110107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:26.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.423+0000 7fb117aaa700 1 --2- 172.21.15.38:0/1258864340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb110071280 0x7fb110107970 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0098e0 tx=0x7fb10c009250 comp rx=0 tx=0).stop 2024-04-22T21:01:26.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.423+0000 7fb117aaa700 1 -- 172.21.15.38:0/1258864340 shutdown_connections 2024-04-22T21:01:26.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.423+0000 7fb117aaa700 1 --2- 172.21.15.38:0/1258864340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb110071280 0x7fb110107970 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:26.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.423+0000 7fb117aaa700 1 -- 172.21.15.38:0/1258864340 >> 172.21.15.38:0/1258864340 conn(0x7fb1100fcf80 msgr2=0x7fb1100ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:26.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.423+0000 7fb117aaa700 1 -- 172.21.15.38:0/1258864340 shutdown_connections 2024-04-22T21:01:26.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.423+0000 7fb117aaa700 1 -- 172.21.15.38:0/1258864340 wait complete. 2024-04-22T21:01:26.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.424+0000 7fb117aaa700 1 Processor -- start 2024-04-22T21:01:26.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.424+0000 7fb117aaa700 1 -- start start 2024-04-22T21:01:26.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.424+0000 7fb117aaa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb110071280 0x7fb110113930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:26.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.424+0000 7fb117aaa700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb110113e70 con 0x7fb110071280 2024-04-22T21:01:26.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.424+0000 7fb115846700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb110071280 0x7fb110113930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:26.380 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.424+0000 7fb115846700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb110071280 0x7fb110113930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42352/0 (socket says 172.21.15.38:42352) 2024-04-22T21:01:26.380 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.424+0000 7fb115846700 1 -- 172.21.15.38:0/4110024529 learned_addr learned my addr 172.21.15.38:0/4110024529 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:26.380 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.424+0000 7fb115846700 1 -- 172.21.15.38:0/4110024529 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10c0094f0 con 0x7fb110071280 2024-04-22T21:01:26.380 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.424+0000 7fb115846700 1 --2- 172.21.15.38:0/4110024529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb110071280 0x7fb110113930 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0094c0 tx=0x7fb10c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:26.380 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.424+0000 7fb1067fc700 1 -- 172.21.15.38:0/4110024529 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb10c0118f0 con 0x7fb110071280 2024-04-22T21:01:26.380 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.425+0000 7fb117aaa700 1 -- 172.21.15.38:0/4110024529 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1101140d0 con 0x7fb110071280 2024-04-22T21:01:26.380 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.425+0000 7fb1067fc700 1 -- 172.21.15.38:0/4110024529 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb10c010d60 con 0x7fb110071280 2024-04-22T21:01:26.380 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.425+0000 7fb1067fc700 1 -- 172.21.15.38:0/4110024529 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb10c024e50 con 0x7fb110071280 2024-04-22T21:01:26.380 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.425+0000 7fb117aaa700 1 -- 172.21.15.38:0/4110024529 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb110114be0 con 0x7fb110071280 2024-04-22T21:01:26.380 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.425+0000 7fb1067fc700 1 -- 172.21.15.38:0/4110024529 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 9) v1 ==== 43389+0+0 (secure 0 0 0) 0x7fb10c01a430 con 0x7fb110071280 2024-04-22T21:01:26.380 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.425+0000 7fb117aaa700 1 -- 172.21.15.38:0/4110024529 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb11005f5f0 con 0x7fb110071280 2024-04-22T21:01:26.380 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.426+0000 7fb1067fc700 1 --2- 172.21.15.38:0/4110024529 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fb0fc036790 0x7fb0fc038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:26.380 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.426+0000 7fb1067fc700 1 -- 172.21.15.38:0/4110024529 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb10c04a630 con 0x7fb110071280 2024-04-22T21:01:26.380 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.426+0000 7fb115045700 1 -- 172.21.15.38:0/4110024529 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fb0fc036790 msgr2=0x7fb0fc038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2551561659 2024-04-22T21:01:26.381 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.426+0000 7fb115045700 1 --2- 172.21.15.38:0/4110024529 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fb0fc036790 0x7fb0fc038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T21:01:26.381 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.430+0000 7fb1067fc700 1 -- 172.21.15.38:0/4110024529 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb10c018350 con 0x7fb110071280 2024-04-22T21:01:26.381 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.627+0000 7fb115045700 1 -- 172.21.15.38:0/4110024529 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fb0fc036790 msgr2=0x7fb0fc038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2551561659 2024-04-22T21:01:26.381 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.627+0000 7fb115045700 1 --2- 172.21.15.38:0/4110024529 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fb0fc036790 0x7fb0fc038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T21:01:26.381 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.726+0000 7fb117aaa700 1 -- 172.21.15.38:0/4110024529 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fb11004bed0 con 0x7fb110071280 2024-04-22T21:01:26.381 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.726+0000 7fb1067fc700 1 -- 172.21.15.38:0/4110024529 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fb10c018350 con 0x7fb110071280 2024-04-22T21:01:26.381 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.728+0000 7fb117aaa700 1 -- 172.21.15.38:0/4110024529 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fb0fc036790 msgr2=0x7fb0fc038c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T21:01:26.381 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.728+0000 7fb117aaa700 1 --2- 172.21.15.38:0/4110024529 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fb0fc036790 0x7fb0fc038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:26.381 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.728+0000 7fb117aaa700 1 -- 172.21.15.38:0/4110024529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb110071280 msgr2=0x7fb110113930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:26.381 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.728+0000 7fb117aaa700 1 --2- 172.21.15.38:0/4110024529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb110071280 0x7fb110113930 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0094c0 tx=0x7fb10c009250 comp rx=0 tx=0).stop 2024-04-22T21:01:26.381 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.728+0000 7fb117aaa700 1 -- 172.21.15.38:0/4110024529 shutdown_connections 2024-04-22T21:01:26.381 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.728+0000 7fb117aaa700 1 --2- 172.21.15.38:0/4110024529 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7fb0fc036790 0x7fb0fc038c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:26.381 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.728+0000 7fb117aaa700 1 --2- 172.21.15.38:0/4110024529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb110071280 0x7fb110113930 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:26.381 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.728+0000 7fb117aaa700 1 -- 172.21.15.38:0/4110024529 >> 172.21.15.38:0/4110024529 conn(0x7fb1100fcf80 msgr2=0x7fb1100fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:26.382 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.728+0000 7fb117aaa700 1 -- 172.21.15.38:0/4110024529 shutdown_connections 2024-04-22T21:01:26.382 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:25.729+0000 7fb117aaa700 1 -- 172.21.15.38:0/4110024529 wait complete. 2024-04-22T21:01:26.382 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for the mgr to restart... 2024-04-22T21:01:26.382 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr epoch 9... 2024-04-22T21:01:31.117 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:31 smithi038 ceph-mon[26374]: Active manager daemon smithi038.lxfgut restarted 2024-04-22T21:01:31.117 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:31 smithi038 ceph-mon[26374]: Activating manager daemon smithi038.lxfgut 2024-04-22T21:01:31.117 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:31 smithi038 ceph-mon[26374]: osdmap e3: 0 total, 0 up, 0 in 2024-04-22T21:01:31.117 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:31 smithi038 ceph-mon[26374]: mgrmap e10: smithi038.lxfgut(active, starting, since 0.0082892s) 2024-04-22T21:01:31.118 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:31 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2024-04-22T21:01:31.118 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:31 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr metadata", "who": "smithi038.lxfgut", "id": "smithi038.lxfgut"}]: dispatch 2024-04-22T21:01:31.118 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:31 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T21:01:31.118 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:31 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T21:01:31.118 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:31 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T21:01:31.118 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:31 smithi038 ceph-mon[26374]: Manager daemon smithi038.lxfgut is now available 2024-04-22T21:01:31.118 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:31 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:01:31.118 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:31 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:01:31.118 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:31 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.lxfgut/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:01:32.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:32 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.lxfgut/trash_purge_schedule"}]: dispatch 2024-04-22T21:01:32.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:32 smithi038 ceph-mon[26374]: mgrmap e11: smithi038.lxfgut(active, since 1.00574s) 2024-04-22T21:01:32.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-22T21:01:32.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-22T21:01:32.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T21:01:32.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-22T21:01:32.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.800+0000 7ff906c80700 1 Processor -- start 2024-04-22T21:01:32.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.800+0000 7ff906c80700 1 -- start start 2024-04-22T21:01:32.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.800+0000 7ff906c80700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9000712c0 0x7ff900105530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:32.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.800+0000 7ff906c80700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9000716d0 con 0x7ff9000712c0 2024-04-22T21:01:32.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.800+0000 7ff904a1c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9000712c0 0x7ff900105530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:32.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.800+0000 7ff904a1c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9000712c0 0x7ff900105530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42366/0 (socket says 172.21.15.38:42366) 2024-04-22T21:01:32.706 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.800+0000 7ff904a1c700 1 -- 172.21.15.38:0/3814833305 learned_addr learned my addr 172.21.15.38:0/3814833305 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:32.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.801+0000 7ff904a1c700 1 -- 172.21.15.38:0/3814833305 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff900105a70 con 0x7ff9000712c0 2024-04-22T21:01:32.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.801+0000 7ff904a1c700 1 --2- 172.21.15.38:0/3814833305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9000712c0 0x7ff900105530 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff8f40098e0 tx=0x7ff8f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd52b5a7cce0ee13 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:32.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.801+0000 7ff8ff7fe700 1 -- 172.21.15.38:0/3814833305 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8f400b860 con 0x7ff9000712c0 2024-04-22T21:01:32.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.801+0000 7ff8ff7fe700 1 -- 172.21.15.38:0/3814833305 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8f400be90 con 0x7ff9000712c0 2024-04-22T21:01:32.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.801+0000 7ff8ff7fe700 1 -- 172.21.15.38:0/3814833305 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8f40103c0 con 0x7ff9000712c0 2024-04-22T21:01:32.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.801+0000 7ff906c80700 1 -- 172.21.15.38:0/3814833305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9000712c0 msgr2=0x7ff900105530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:32.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.801+0000 7ff906c80700 1 --2- 172.21.15.38:0/3814833305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9000712c0 0x7ff900105530 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff8f40098e0 tx=0x7ff8f4009250 comp rx=0 tx=0).stop 2024-04-22T21:01:32.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.802+0000 7ff906c80700 1 -- 172.21.15.38:0/3814833305 shutdown_connections 2024-04-22T21:01:32.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.802+0000 7ff906c80700 1 --2- 172.21.15.38:0/3814833305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9000712c0 0x7ff900105530 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:32.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.802+0000 7ff906c80700 1 -- 172.21.15.38:0/3814833305 >> 172.21.15.38:0/3814833305 conn(0x7ff9000fcf80 msgr2=0x7ff9000ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:32.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.802+0000 7ff906c80700 1 -- 172.21.15.38:0/3814833305 shutdown_connections 2024-04-22T21:01:32.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.802+0000 7ff906c80700 1 -- 172.21.15.38:0/3814833305 wait complete. 2024-04-22T21:01:32.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.802+0000 7ff906c80700 1 Processor -- start 2024-04-22T21:01:32.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.802+0000 7ff906c80700 1 -- start start 2024-04-22T21:01:32.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.802+0000 7ff906c80700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9000712c0 0x7ff900111790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:32.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.802+0000 7ff906c80700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff900111cd0 con 0x7ff9000712c0 2024-04-22T21:01:32.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.803+0000 7ff904a1c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9000712c0 0x7ff900111790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:32.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.803+0000 7ff904a1c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9000712c0 0x7ff900111790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42376/0 (socket says 172.21.15.38:42376) 2024-04-22T21:01:32.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.803+0000 7ff904a1c700 1 -- 172.21.15.38:0/3173131707 learned_addr learned my addr 172.21.15.38:0/3173131707 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:32.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.803+0000 7ff904a1c700 1 -- 172.21.15.38:0/3173131707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f40094f0 con 0x7ff9000712c0 2024-04-22T21:01:32.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.803+0000 7ff904a1c700 1 --2- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9000712c0 0x7ff900111790 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff8f40105b0 tx=0x7ff8f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:32.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.803+0000 7ff8fd7fa700 1 -- 172.21.15.38:0/3173131707 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8f4010840 con 0x7ff9000712c0 2024-04-22T21:01:32.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.803+0000 7ff906c80700 1 -- 172.21.15.38:0/3173131707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff900111ed0 con 0x7ff9000712c0 2024-04-22T21:01:32.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.803+0000 7ff8fd7fa700 1 -- 172.21.15.38:0/3173131707 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8f4010e70 con 0x7ff9000712c0 2024-04-22T21:01:32.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.803+0000 7ff8fd7fa700 1 -- 172.21.15.38:0/3173131707 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8f401ae30 con 0x7ff9000712c0 2024-04-22T21:01:32.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.803+0000 7ff906c80700 1 -- 172.21.15.38:0/3173131707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff900112a40 con 0x7ff9000712c0 2024-04-22T21:01:32.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.804+0000 7ff8fd7fa700 1 -- 172.21.15.38:0/3173131707 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 9) v1 ==== 43389+0+0 (secure 0 0 0) 0x7ff8f402d3f0 con 0x7ff9000712c0 2024-04-22T21:01:32.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.804+0000 7ff8fd7fa700 1 --2- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7ff8f00367e0 0x7ff8f0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:32.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.804+0000 7ff8fd7fa700 1 -- 172.21.15.38:0/3173131707 --> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8f00393a0 con 0x7ff8f00367e0 2024-04-22T21:01:32.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.804+0000 7ff8fd7fa700 1 -- 172.21.15.38:0/3173131707 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff8f404a370 con 0x7ff9000712c0 2024-04-22T21:01:32.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.804+0000 7ff8fffff700 1 -- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7ff8f00367e0 msgr2=0x7ff8f0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2551561659 2024-04-22T21:01:32.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:26.804+0000 7ff8fffff700 1 --2- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7ff8f00367e0 0x7ff8f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T21:01:32.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:27.005+0000 7ff8fffff700 1 -- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7ff8f00367e0 msgr2=0x7ff8f0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2551561659 2024-04-22T21:01:32.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:27.005+0000 7ff8fffff700 1 --2- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7ff8f00367e0 0x7ff8f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T21:01:32.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:27.406+0000 7ff8fffff700 1 -- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7ff8f00367e0 msgr2=0x7ff8f0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2551561659 2024-04-22T21:01:32.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:27.406+0000 7ff8fffff700 1 --2- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7ff8f00367e0 0x7ff8f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T21:01:32.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:28.207+0000 7ff8fffff700 1 -- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7ff8f00367e0 msgr2=0x7ff8f0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2551561659 2024-04-22T21:01:32.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:28.207+0000 7ff8fffff700 1 --2- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7ff8f00367e0 0x7ff8f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T21:01:32.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:29.808+0000 7ff8fffff700 1 -- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7ff8f00367e0 msgr2=0x7ff8f0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2551561659 2024-04-22T21:01:32.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:29.808+0000 7ff8fffff700 1 --2- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7ff8f00367e0 0x7ff8f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T21:01:32.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:31.051+0000 7ff8fd7fa700 1 -- 172.21.15.38:0/3173131707 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mgrmap(e 10) v1 ==== 43156+0+0 (secure 0 0 0) 0x7ff8f401a430 con 0x7ff9000712c0 2024-04-22T21:01:32.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:31.051+0000 7ff8fd7fa700 1 -- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7ff8f00367e0 msgr2=0x7ff8f0038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T21:01:32.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:31.051+0000 7ff8fd7fa700 1 --2- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7ff8f00367e0 0x7ff8f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:32.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.047+0000 7ff8fd7fa700 1 -- 172.21.15.38:0/3173131707 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 11) v1 ==== 43283+0+0 (secure 0 0 0) 0x7ff8f4018350 con 0x7ff9000712c0 2024-04-22T21:01:32.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.047+0000 7ff8fd7fa700 1 --2- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7ff8f003a070 0x7ff8f003c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:32.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.047+0000 7ff8fd7fa700 1 -- 172.21.15.38:0/3173131707 --> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8f00393a0 con 0x7ff8f003a070 2024-04-22T21:01:32.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.048+0000 7ff8fffff700 1 --2- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7ff8f003a070 0x7ff8f003c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:32.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.049+0000 7ff8fffff700 1 --2- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7ff8f003a070 0x7ff8f003c450 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec0039d0 tx=0x7ff8ec0092f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:32.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.049+0000 7ff8fd7fa700 1 -- 172.21.15.38:0/3173131707 <== mgr.14162 v2:172.21.15.38:6800/989134174 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7ff8f00393a0 con 0x7ff8f003a070 2024-04-22T21:01:32.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.060+0000 7ff906c80700 1 -- 172.21.15.38:0/3173131707 --> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff900112360 con 0x7ff8f003a070 2024-04-22T21:01:32.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.060+0000 7ff8fd7fa700 1 -- 172.21.15.38:0/3173131707 <== mgr.14162 v2:172.21.15.38:6800/989134174 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7ff900112360 con 0x7ff8f003a070 2024-04-22T21:01:32.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.060+0000 7ff906c80700 1 -- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7ff8f003a070 msgr2=0x7ff8f003c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:32.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.060+0000 7ff906c80700 1 --2- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7ff8f003a070 0x7ff8f003c450 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec0039d0 tx=0x7ff8ec0092f0 comp rx=0 tx=0).stop 2024-04-22T21:01:32.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.060+0000 7ff906c80700 1 -- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9000712c0 msgr2=0x7ff900111790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:32.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.060+0000 7ff906c80700 1 --2- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9000712c0 0x7ff900111790 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff8f40105b0 tx=0x7ff8f4009250 comp rx=0 tx=0).stop 2024-04-22T21:01:32.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.060+0000 7ff906c80700 1 -- 172.21.15.38:0/3173131707 shutdown_connections 2024-04-22T21:01:32.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.060+0000 7ff906c80700 1 --2- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7ff8f003a070 0x7ff8f003c450 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:32.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.060+0000 7ff906c80700 1 --2- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9000712c0 0x7ff900111790 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:32.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.060+0000 7ff906c80700 1 --2- 172.21.15.38:0/3173131707 >> [v2:172.21.15.38:6800/2551561659,v1:172.21.15.38:6801/2551561659] conn(0x7ff8f00367e0 0x7ff8f0038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:32.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.060+0000 7ff906c80700 1 -- 172.21.15.38:0/3173131707 >> 172.21.15.38:0/3173131707 conn(0x7ff9000fcf80 msgr2=0x7ff9000fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:32.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.061+0000 7ff906c80700 1 -- 172.21.15.38:0/3173131707 shutdown_connections 2024-04-22T21:01:32.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:32.061+0000 7ff906c80700 1 -- 172.21.15.38:0/3173131707 wait complete. 2024-04-22T21:01:32.711 INFO:teuthology.orchestra.run.smithi038.stdout:mgr epoch 9 is available 2024-04-22T21:01:32.711 INFO:teuthology.orchestra.run.smithi038.stdout:Generating a dashboard self-signed certificate... 2024-04-22T21:01:33.372 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:33 smithi038 ceph-mon[26374]: [22/Apr/2024:21:01:32] ENGINE Bus STARTING 2024-04-22T21:01:33.372 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:33 smithi038 ceph-mon[26374]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T21:01:33.373 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:33 smithi038 ceph-mon[26374]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T21:01:33.373 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:33 smithi038 ceph-mon[26374]: [22/Apr/2024:21:01:32] ENGINE Serving on https://172.21.15.38:7150 2024-04-22T21:01:33.373 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:33 smithi038 ceph-mon[26374]: [22/Apr/2024:21:01:32] ENGINE Bus STARTED 2024-04-22T21:01:33.373 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:33 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:33.373 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:33 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:34.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-22T21:01:34.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.151+0000 7fa2f186e700 1 Processor -- start 2024-04-22T21:01:34.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.151+0000 7fa2f186e700 1 -- start start 2024-04-22T21:01:34.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.151+0000 7fa2f186e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2ec101890 0x7fa2ec101ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:34.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.151+0000 7fa2f186e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec1021e0 con 0x7fa2ec101890 2024-04-22T21:01:34.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.151+0000 7fa2eaffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2ec101890 0x7fa2ec101ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:34.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.151+0000 7fa2eaffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2ec101890 0x7fa2ec101ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46642/0 (socket says 172.21.15.38:46642) 2024-04-22T21:01:34.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.151+0000 7fa2eaffd700 1 -- 172.21.15.38:0/2333199489 learned_addr learned my addr 172.21.15.38:0/2333199489 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:34.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.152+0000 7fa2eaffd700 1 -- 172.21.15.38:0/2333199489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec102320 con 0x7fa2ec101890 2024-04-22T21:01:34.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.152+0000 7fa2eaffd700 1 --2- 172.21.15.38:0/2333199489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2ec101890 0x7fa2ec101ca0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc00bb80 tx=0x7fa2dc00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc72a99a662a2e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:34.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.152+0000 7fa2e9ffb700 1 -- 172.21.15.38:0/2333199489 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2dc00dd90 con 0x7fa2ec101890 2024-04-22T21:01:34.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.152+0000 7fa2e9ffb700 1 -- 172.21.15.38:0/2333199489 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2dc010940 con 0x7fa2ec101890 2024-04-22T21:01:34.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.152+0000 7fa2e9ffb700 1 -- 172.21.15.38:0/2333199489 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2dc0189b0 con 0x7fa2ec101890 2024-04-22T21:01:34.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.153+0000 7fa2f186e700 1 -- 172.21.15.38:0/2333199489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2ec101890 msgr2=0x7fa2ec101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:34.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.153+0000 7fa2f186e700 1 --2- 172.21.15.38:0/2333199489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2ec101890 0x7fa2ec101ca0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc00bb80 tx=0x7fa2dc00b4f0 comp rx=0 tx=0).stop 2024-04-22T21:01:34.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.153+0000 7fa2f186e700 1 -- 172.21.15.38:0/2333199489 shutdown_connections 2024-04-22T21:01:34.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.153+0000 7fa2f186e700 1 --2- 172.21.15.38:0/2333199489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2ec101890 0x7fa2ec101ca0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:34.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.153+0000 7fa2f186e700 1 -- 172.21.15.38:0/2333199489 >> 172.21.15.38:0/2333199489 conn(0x7fa2ec0fd050 msgr2=0x7fa2ec0ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:34.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.153+0000 7fa2f186e700 1 -- 172.21.15.38:0/2333199489 shutdown_connections 2024-04-22T21:01:34.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.153+0000 7fa2f186e700 1 -- 172.21.15.38:0/2333199489 wait complete. 2024-04-22T21:01:34.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.153+0000 7fa2f186e700 1 Processor -- start 2024-04-22T21:01:34.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.154+0000 7fa2f186e700 1 -- start start 2024-04-22T21:01:34.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.154+0000 7fa2f186e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2ec101890 0x7fa2ec10f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:34.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.154+0000 7fa2f186e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec10f9f0 con 0x7fa2ec101890 2024-04-22T21:01:34.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.154+0000 7fa2eaffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2ec101890 0x7fa2ec10f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:34.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.154+0000 7fa2eaffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2ec101890 0x7fa2ec10f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46646/0 (socket says 172.21.15.38:46646) 2024-04-22T21:01:34.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.154+0000 7fa2eaffd700 1 -- 172.21.15.38:0/782362485 learned_addr learned my addr 172.21.15.38:0/782362485 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:34.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.154+0000 7fa2eaffd700 1 -- 172.21.15.38:0/782362485 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2dc00b790 con 0x7fa2ec101890 2024-04-22T21:01:34.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.154+0000 7fa2eaffd700 1 --2- 172.21.15.38:0/782362485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2ec101890 0x7fa2ec10f4b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc006b00 tx=0x7fa2dc00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:34.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.154+0000 7fa2d37fe700 1 -- 172.21.15.38:0/782362485 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2dc010c10 con 0x7fa2ec101890 2024-04-22T21:01:34.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.154+0000 7fa2d37fe700 1 -- 172.21.15.38:0/782362485 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2dc01d050 con 0x7fa2ec101890 2024-04-22T21:01:34.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.155+0000 7fa2d37fe700 1 -- 172.21.15.38:0/782362485 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2dc0226b0 con 0x7fa2ec101890 2024-04-22T21:01:34.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.155+0000 7fa2f186e700 1 -- 172.21.15.38:0/782362485 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ec10fbf0 con 0x7fa2ec101890 2024-04-22T21:01:34.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.155+0000 7fa2f186e700 1 -- 172.21.15.38:0/782362485 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ec110760 con 0x7fa2ec101890 2024-04-22T21:01:34.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.155+0000 7fa2d37fe700 1 -- 172.21.15.38:0/782362485 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 11) v1 ==== 43283+0+0 (secure 0 0 0) 0x7fa2dc022810 con 0x7fa2ec101890 2024-04-22T21:01:34.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.156+0000 7fa2d37fe700 1 --2- 172.21.15.38:0/782362485 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fa2d4036710 0x7fa2d4038bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:34.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.156+0000 7fa2f186e700 1 -- 172.21.15.38:0/782362485 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ec05f5f0 con 0x7fa2ec101890 2024-04-22T21:01:34.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.156+0000 7fa2d37fe700 1 -- 172.21.15.38:0/782362485 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa2dc014070 con 0x7fa2ec101890 2024-04-22T21:01:34.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.156+0000 7fa2ea7fc700 1 --2- 172.21.15.38:0/782362485 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fa2d4036710 0x7fa2d4038bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:34.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.156+0000 7fa2ea7fc700 1 --2- 172.21.15.38:0/782362485 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fa2d4036710 0x7fa2d4038bc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0009990 tx=0x7fa2e0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:34.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.160+0000 7fa2d37fe700 1 -- 172.21.15.38:0/782362485 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa2dc021510 con 0x7fa2ec101890 2024-04-22T21:01:34.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.375+0000 7fa2f186e700 1 -- 172.21.15.38:0/782362485 --> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2ec110b10 con 0x7fa2d4036710 2024-04-22T21:01:34.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.477+0000 7fa2d37fe700 1 -- 172.21.15.38:0/782362485 <== mgr.14162 v2:172.21.15.38:6800/989134174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fa2ec110b10 con 0x7fa2d4036710 2024-04-22T21:01:34.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.479+0000 7fa2f186e700 1 -- 172.21.15.38:0/782362485 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fa2d4036710 msgr2=0x7fa2d4038bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:34.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.479+0000 7fa2f186e700 1 --2- 172.21.15.38:0/782362485 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fa2d4036710 0x7fa2d4038bc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0009990 tx=0x7fa2e0006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:34.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.479+0000 7fa2f186e700 1 -- 172.21.15.38:0/782362485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2ec101890 msgr2=0x7fa2ec10f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:34.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.479+0000 7fa2f186e700 1 --2- 172.21.15.38:0/782362485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2ec101890 0x7fa2ec10f4b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc006b00 tx=0x7fa2dc00b4f0 comp rx=0 tx=0).stop 2024-04-22T21:01:34.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.479+0000 7fa2f186e700 1 -- 172.21.15.38:0/782362485 shutdown_connections 2024-04-22T21:01:34.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.479+0000 7fa2f186e700 1 --2- 172.21.15.38:0/782362485 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fa2d4036710 0x7fa2d4038bc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:34.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.479+0000 7fa2f186e700 1 --2- 172.21.15.38:0/782362485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2ec101890 0x7fa2ec10f4b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:34.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.479+0000 7fa2f186e700 1 -- 172.21.15.38:0/782362485 >> 172.21.15.38:0/782362485 conn(0x7fa2ec0fd050 msgr2=0x7fa2ec105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:34.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.479+0000 7fa2f186e700 1 -- 172.21.15.38:0/782362485 shutdown_connections 2024-04-22T21:01:34.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:33.479+0000 7fa2f186e700 1 -- 172.21.15.38:0/782362485 wait complete. 2024-04-22T21:01:34.057 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial admin user... 2024-04-22T21:01:34.694 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:34 smithi038 ceph-mon[26374]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:01:34.694 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:34 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:34.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:34 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:34.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:34 smithi038 ceph-mon[26374]: mgrmap e12: smithi038.lxfgut(active, since 2s) 2024-04-22T21:01:34.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:34 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:35.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$8iCyds8mDbFZKhL8/.akau2RYAooktrovG/4H6lb/uZz8ttPsMGOO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713819694, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-22T21:01:35.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.465+0000 7f224856e700 1 Processor -- start 2024-04-22T21:01:35.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.465+0000 7f224856e700 1 -- start start 2024-04-22T21:01:35.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.465+0000 7f224856e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2240102940 0x7f2240102d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:35.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.465+0000 7f224856e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2240103290 con 0x7f2240102940 2024-04-22T21:01:35.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.465+0000 7f224630a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2240102940 0x7f2240102d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:35.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.465+0000 7f224630a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2240102940 0x7f2240102d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46656/0 (socket says 172.21.15.38:46656) 2024-04-22T21:01:35.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.465+0000 7f224630a700 1 -- 172.21.15.38:0/713689016 learned_addr learned my addr 172.21.15.38:0/713689016 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:35.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.466+0000 7f224630a700 1 -- 172.21.15.38:0/713689016 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22401033d0 con 0x7f2240102940 2024-04-22T21:01:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.466+0000 7f224630a700 1 --2- 172.21.15.38:0/713689016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2240102940 0x7f2240102d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f223c0098e0 tx=0x7f223c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ed36438d57fd45 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.466+0000 7f2245308700 1 -- 172.21.15.38:0/713689016 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f223c00b8f0 con 0x7f2240102940 2024-04-22T21:01:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.466+0000 7f2245308700 1 -- 172.21.15.38:0/713689016 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f223c011420 con 0x7f2240102940 2024-04-22T21:01:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.466+0000 7f2245308700 1 -- 172.21.15.38:0/713689016 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f223c010540 con 0x7f2240102940 2024-04-22T21:01:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.467+0000 7f224856e700 1 -- 172.21.15.38:0/713689016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2240102940 msgr2=0x7f2240102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.467+0000 7f224856e700 1 --2- 172.21.15.38:0/713689016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2240102940 0x7f2240102d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f223c0098e0 tx=0x7f223c009250 comp rx=0 tx=0).stop 2024-04-22T21:01:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.467+0000 7f224856e700 1 -- 172.21.15.38:0/713689016 shutdown_connections 2024-04-22T21:01:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.467+0000 7f224856e700 1 --2- 172.21.15.38:0/713689016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2240102940 0x7f2240102d50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.467+0000 7f224856e700 1 -- 172.21.15.38:0/713689016 >> 172.21.15.38:0/713689016 conn(0x7f22400fe100 msgr2=0x7f2240100510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.467+0000 7f224856e700 1 -- 172.21.15.38:0/713689016 shutdown_connections 2024-04-22T21:01:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.467+0000 7f224856e700 1 -- 172.21.15.38:0/713689016 wait complete. 2024-04-22T21:01:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.467+0000 7f224856e700 1 Processor -- start 2024-04-22T21:01:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.467+0000 7f224856e700 1 -- start start 2024-04-22T21:01:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.468+0000 7f224856e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2240102940 0x7f2240078c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.468+0000 7f224856e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2240079190 con 0x7f2240102940 2024-04-22T21:01:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.468+0000 7f224630a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2240102940 0x7f2240078c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.468+0000 7f224630a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2240102940 0x7f2240078c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46670/0 (socket says 172.21.15.38:46670) 2024-04-22T21:01:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.468+0000 7f224630a700 1 -- 172.21.15.38:0/174565352 learned_addr learned my addr 172.21.15.38:0/174565352 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.468+0000 7f224630a700 1 -- 172.21.15.38:0/174565352 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c0094f0 con 0x7f2240102940 2024-04-22T21:01:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.468+0000 7f224630a700 1 --2- 172.21.15.38:0/174565352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2240102940 0x7f2240078c50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f223c0090b0 tx=0x7f223c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.469+0000 7f2236ffd700 1 -- 172.21.15.38:0/174565352 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f223c0107b0 con 0x7f2240102940 2024-04-22T21:01:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.469+0000 7f2236ffd700 1 -- 172.21.15.38:0/174565352 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f223c010de0 con 0x7f2240102940 2024-04-22T21:01:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.469+0000 7f224856e700 1 -- 172.21.15.38:0/174565352 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2240077200 con 0x7f2240102940 2024-04-22T21:01:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.469+0000 7f2236ffd700 1 -- 172.21.15.38:0/174565352 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f223c01ae50 con 0x7f2240102940 2024-04-22T21:01:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.469+0000 7f224856e700 1 -- 172.21.15.38:0/174565352 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22400776f0 con 0x7f2240102940 2024-04-22T21:01:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.470+0000 7f2236ffd700 1 -- 172.21.15.38:0/174565352 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) v1 ==== 43389+0+0 (secure 0 0 0) 0x7f223c02d3f0 con 0x7f2240102940 2024-04-22T21:01:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.470+0000 7f224856e700 1 -- 172.21.15.38:0/174565352 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224004bed0 con 0x7f2240102940 2024-04-22T21:01:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.470+0000 7f2236ffd700 1 --2- 172.21.15.38:0/174565352 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f222c036870 0x7f222c038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.470+0000 7f2236ffd700 1 -- 172.21.15.38:0/174565352 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f223c04a230 con 0x7f2240102940 2024-04-22T21:01:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.470+0000 7f2245b09700 1 --2- 172.21.15.38:0/174565352 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f222c036870 0x7f222c038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.470+0000 7f2245b09700 1 --2- 172.21.15.38:0/174565352 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f222c036870 0x7f222c038d20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2230009990 tx=0x7f2230006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.475+0000 7f2236ffd700 1 -- 172.21.15.38:0/174565352 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f223c02b8e0 con 0x7f2240102940 2024-04-22T21:01:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.697+0000 7f224856e700 1 -- 172.21.15.38:0/174565352 --> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f224005f410 con 0x7f222c036870 2024-04-22T21:01:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.945+0000 7f2236ffd700 1 -- 172.21.15.38:0/174565352 <== mgr.14162 v2:172.21.15.38:6800/989134174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f224005f410 con 0x7f222c036870 2024-04-22T21:01:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.947+0000 7f224856e700 1 -- 172.21.15.38:0/174565352 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f222c036870 msgr2=0x7f222c038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.947+0000 7f224856e700 1 --2- 172.21.15.38:0/174565352 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f222c036870 0x7f222c038d20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2230009990 tx=0x7f2230006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.947+0000 7f224856e700 1 -- 172.21.15.38:0/174565352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2240102940 msgr2=0x7f2240078c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.947+0000 7f224856e700 1 --2- 172.21.15.38:0/174565352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2240102940 0x7f2240078c50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f223c0090b0 tx=0x7f223c009250 comp rx=0 tx=0).stop 2024-04-22T21:01:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.947+0000 7f224856e700 1 -- 172.21.15.38:0/174565352 shutdown_connections 2024-04-22T21:01:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.947+0000 7f224856e700 1 --2- 172.21.15.38:0/174565352 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f222c036870 0x7f222c038d20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.947+0000 7f224856e700 1 --2- 172.21.15.38:0/174565352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2240102940 0x7f2240078c50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.947+0000 7f224856e700 1 -- 172.21.15.38:0/174565352 >> 172.21.15.38:0/174565352 conn(0x7f22400fe100 msgr2=0x7f22400febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.947+0000 7f224856e700 1 -- 172.21.15.38:0/174565352 shutdown_connections 2024-04-22T21:01:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:34.947+0000 7f224856e700 1 -- 172.21.15.38:0/174565352 wait complete. 2024-04-22T21:01:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:Fetching dashboard port number... 2024-04-22T21:01:36.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:35 smithi038 ceph-mon[26374]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:01:36.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:35 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:37.056 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:36 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1785366066' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-22T21:01:37.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 8443 2024-04-22T21:01:37.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.548+0000 7f08503e5700 1 Processor -- start 2024-04-22T21:01:37.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.548+0000 7f08503e5700 1 -- start start 2024-04-22T21:01:37.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.548+0000 7f08503e5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0848102930 0x7f0848102d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:37.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.548+0000 7f08503e5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0848103280 con 0x7f0848102930 2024-04-22T21:01:37.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.548+0000 7f084e181700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0848102930 0x7f0848102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:37.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.548+0000 7f084e181700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0848102930 0x7f0848102d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46676/0 (socket says 172.21.15.38:46676) 2024-04-22T21:01:37.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.548+0000 7f084e181700 1 -- 172.21.15.38:0/719713916 learned_addr learned my addr 172.21.15.38:0/719713916 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:37.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.548+0000 7f084e181700 1 -- 172.21.15.38:0/719713916 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08481033c0 con 0x7f0848102930 2024-04-22T21:01:37.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.549+0000 7f084e181700 1 --2- 172.21.15.38:0/719713916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0848102930 0x7f0848102d40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f08440098e0 tx=0x7f0844009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796fa95c3df30f0c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:37.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.549+0000 7f084d17f700 1 -- 172.21.15.38:0/719713916 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f084400b8f0 con 0x7f0848102930 2024-04-22T21:01:37.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.549+0000 7f084d17f700 1 -- 172.21.15.38:0/719713916 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0844011420 con 0x7f0848102930 2024-04-22T21:01:37.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.549+0000 7f084d17f700 1 -- 172.21.15.38:0/719713916 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0844010540 con 0x7f0848102930 2024-04-22T21:01:37.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.549+0000 7f08503e5700 1 -- 172.21.15.38:0/719713916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0848102930 msgr2=0x7f0848102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:37.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.549+0000 7f08503e5700 1 --2- 172.21.15.38:0/719713916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0848102930 0x7f0848102d40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f08440098e0 tx=0x7f0844009250 comp rx=0 tx=0).stop 2024-04-22T21:01:37.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.549+0000 7f08503e5700 1 -- 172.21.15.38:0/719713916 shutdown_connections 2024-04-22T21:01:37.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.549+0000 7f08503e5700 1 --2- 172.21.15.38:0/719713916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0848102930 0x7f0848102d40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:37.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.549+0000 7f08503e5700 1 -- 172.21.15.38:0/719713916 >> 172.21.15.38:0/719713916 conn(0x7f08480fe140 msgr2=0x7f0848100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:37.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.550+0000 7f08503e5700 1 -- 172.21.15.38:0/719713916 shutdown_connections 2024-04-22T21:01:37.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.550+0000 7f08503e5700 1 -- 172.21.15.38:0/719713916 wait complete. 2024-04-22T21:01:37.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.550+0000 7f08503e5700 1 Processor -- start 2024-04-22T21:01:37.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.550+0000 7f08503e5700 1 -- start start 2024-04-22T21:01:37.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.550+0000 7f08503e5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0848102930 0x7f0848111720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:37.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.550+0000 7f08503e5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0848111c60 con 0x7f0848102930 2024-04-22T21:01:37.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.550+0000 7f084e181700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0848102930 0x7f0848111720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:37.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.550+0000 7f084e181700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0848102930 0x7f0848111720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46680/0 (socket says 172.21.15.38:46680) 2024-04-22T21:01:37.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.550+0000 7f084e181700 1 -- 172.21.15.38:0/1785366066 learned_addr learned my addr 172.21.15.38:0/1785366066 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:37.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.551+0000 7f084e181700 1 -- 172.21.15.38:0/1785366066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08440094f0 con 0x7f0848102930 2024-04-22T21:01:37.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.551+0000 7f084e181700 1 --2- 172.21.15.38:0/1785366066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0848102930 0x7f0848111720 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0844006b00 tx=0x7f0844009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:37.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.551+0000 7f083effd700 1 -- 172.21.15.38:0/1785366066 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0844010800 con 0x7f0848102930 2024-04-22T21:01:37.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.551+0000 7f083effd700 1 -- 172.21.15.38:0/1785366066 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0844010e30 con 0x7f0848102930 2024-04-22T21:01:37.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.551+0000 7f083effd700 1 -- 172.21.15.38:0/1785366066 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f084401ae40 con 0x7f0848102930 2024-04-22T21:01:37.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.551+0000 7f08503e5700 1 -- 172.21.15.38:0/1785366066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0848111e60 con 0x7f0848102930 2024-04-22T21:01:37.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.551+0000 7f08503e5700 1 -- 172.21.15.38:0/1785366066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08481129d0 con 0x7f0848102930 2024-04-22T21:01:37.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.552+0000 7f083effd700 1 -- 172.21.15.38:0/1785366066 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) v1 ==== 43389+0+0 (secure 0 0 0) 0x7f084402e440 con 0x7f0848102930 2024-04-22T21:01:37.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.552+0000 7f083effd700 1 --2- 172.21.15.38:0/1785366066 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f08340367e0 0x7f0834038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:37.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.552+0000 7f083effd700 1 -- 172.21.15.38:0/1785366066 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f084404a6d0 con 0x7f0848102930 2024-04-22T21:01:37.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.552+0000 7f084d980700 1 --2- 172.21.15.38:0/1785366066 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f08340367e0 0x7f0834038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:37.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.552+0000 7f08503e5700 1 -- 172.21.15.38:0/1785366066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084805f5f0 con 0x7f0848102930 2024-04-22T21:01:37.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.553+0000 7f084d980700 1 --2- 172.21.15.38:0/1785366066 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f08340367e0 0x7f0834038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0838009990 tx=0x7f0838006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:37.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.557+0000 7f083effd700 1 -- 172.21.15.38:0/1785366066 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f0844018530 con 0x7f0848102930 2024-04-22T21:01:37.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.752+0000 7f08503e5700 1 -- 172.21.15.38:0/1785366066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f084804bed0 con 0x7f0848102930 2024-04-22T21:01:37.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.753+0000 7f083effd700 1 -- 172.21.15.38:0/1785366066 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f0844048030 con 0x7f0848102930 2024-04-22T21:01:37.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.755+0000 7f08503e5700 1 -- 172.21.15.38:0/1785366066 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f08340367e0 msgr2=0x7f0834038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:37.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.755+0000 7f08503e5700 1 --2- 172.21.15.38:0/1785366066 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f08340367e0 0x7f0834038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0838009990 tx=0x7f0838006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:37.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.755+0000 7f08503e5700 1 -- 172.21.15.38:0/1785366066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0848102930 msgr2=0x7f0848111720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:37.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.755+0000 7f08503e5700 1 --2- 172.21.15.38:0/1785366066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0848102930 0x7f0848111720 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0844006b00 tx=0x7f0844009250 comp rx=0 tx=0).stop 2024-04-22T21:01:37.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.755+0000 7f08503e5700 1 -- 172.21.15.38:0/1785366066 shutdown_connections 2024-04-22T21:01:37.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.755+0000 7f08503e5700 1 --2- 172.21.15.38:0/1785366066 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f08340367e0 0x7f0834038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:37.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.755+0000 7f08503e5700 1 --2- 172.21.15.38:0/1785366066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0848102930 0x7f0848111720 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:37.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.755+0000 7f08503e5700 1 -- 172.21.15.38:0/1785366066 >> 172.21.15.38:0/1785366066 conn(0x7f08480fe140 msgr2=0x7f08480fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:37.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.755+0000 7f08503e5700 1 -- 172.21.15.38:0/1785366066 shutdown_connections 2024-04-22T21:01:37.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:36.755+0000 7f08503e5700 1 -- 172.21.15.38:0/1785366066 wait complete. 2024-04-22T21:01:37.357 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T21:01:37.357 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-04-22T21:01:37.365 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T21:01:37.366 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-22T21:01:37.366 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-04-22T21:01:37.366 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-22T21:01:37.368 INFO:teuthology.orchestra.run.smithi038.stdout:Ceph Dashboard is now available at: 2024-04-22T21:01:37.368 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:01:37.368 INFO:teuthology.orchestra.run.smithi038.stdout: URL: https://smithi038:8443/ 2024-04-22T21:01:37.368 INFO:teuthology.orchestra.run.smithi038.stdout: User: admin 2024-04-22T21:01:37.368 INFO:teuthology.orchestra.run.smithi038.stdout: Password: mjs4reccyy 2024-04-22T21:01:37.368 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:01:37.368 INFO:teuthology.orchestra.run.smithi038.stdout:Saving cluster configuration to /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/config directory 2024-04-22T21:01:37.368 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling autotune for osd_memory_target 2024-04-22T21:01:38.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.826+0000 7f0223660700 1 Processor -- start 2024-04-22T21:01:38.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.826+0000 7f0223660700 1 -- start start 2024-04-22T21:01:38.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.826+0000 7f0223660700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f021c102910 0x7f021c102d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:38.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.827+0000 7f0223660700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c103260 con 0x7f021c102910 2024-04-22T21:01:38.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.827+0000 7f02213fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f021c102910 0x7f021c102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:38.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.827+0000 7f02213fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f021c102910 0x7f021c102d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46696/0 (socket says 172.21.15.38:46696) 2024-04-22T21:01:38.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.827+0000 7f02213fc700 1 -- 172.21.15.38:0/951985737 learned_addr learned my addr 172.21.15.38:0/951985737 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:38.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.827+0000 7f02213fc700 1 -- 172.21.15.38:0/951985737 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c1033a0 con 0x7f021c102910 2024-04-22T21:01:38.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.827+0000 7f02213fc700 1 --2- 172.21.15.38:0/951985737 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f021c102910 0x7f021c102d20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f02180098e0 tx=0x7f0218009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee8952c1390e759a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:38.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.828+0000 7f0213fff700 1 -- 172.21.15.38:0/951985737 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f021800b860 con 0x7f021c102910 2024-04-22T21:01:38.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.828+0000 7f0213fff700 1 -- 172.21.15.38:0/951985737 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f021800be90 con 0x7f021c102910 2024-04-22T21:01:38.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.828+0000 7f0213fff700 1 -- 172.21.15.38:0/951985737 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02180103c0 con 0x7f021c102910 2024-04-22T21:01:38.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.828+0000 7f0223660700 1 -- 172.21.15.38:0/951985737 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f021c102910 msgr2=0x7f021c102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:38.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.828+0000 7f0223660700 1 --2- 172.21.15.38:0/951985737 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f021c102910 0x7f021c102d20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f02180098e0 tx=0x7f0218009250 comp rx=0 tx=0).stop 2024-04-22T21:01:38.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.828+0000 7f0223660700 1 -- 172.21.15.38:0/951985737 shutdown_connections 2024-04-22T21:01:38.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.828+0000 7f0223660700 1 --2- 172.21.15.38:0/951985737 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f021c102910 0x7f021c102d20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:38.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.828+0000 7f0223660700 1 -- 172.21.15.38:0/951985737 >> 172.21.15.38:0/951985737 conn(0x7f021c0fe0b0 msgr2=0x7f021c1004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:38.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.828+0000 7f0223660700 1 -- 172.21.15.38:0/951985737 shutdown_connections 2024-04-22T21:01:38.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.828+0000 7f0223660700 1 -- 172.21.15.38:0/951985737 wait complete. 2024-04-22T21:01:38.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.829+0000 7f0223660700 1 Processor -- start 2024-04-22T21:01:38.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.829+0000 7f0223660700 1 -- start start 2024-04-22T21:01:38.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.829+0000 7f0223660700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f021c102910 0x7f021c111750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:38.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.829+0000 7f0223660700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c111c90 con 0x7f021c102910 2024-04-22T21:01:38.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.829+0000 7f02213fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f021c102910 0x7f021c111750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:38.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.829+0000 7f02213fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f021c102910 0x7f021c111750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46712/0 (socket says 172.21.15.38:46712) 2024-04-22T21:01:38.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.829+0000 7f02213fc700 1 -- 172.21.15.38:0/4289231869 learned_addr learned my addr 172.21.15.38:0/4289231869 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:38.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.830+0000 7f02213fc700 1 -- 172.21.15.38:0/4289231869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02180094f0 con 0x7f021c102910 2024-04-22T21:01:38.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.830+0000 7f02213fc700 1 --2- 172.21.15.38:0/4289231869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f021c102910 0x7f021c111750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0218006b00 tx=0x7f0218009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:38.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.830+0000 7f0211ffb700 1 -- 172.21.15.38:0/4289231869 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0218010840 con 0x7f021c102910 2024-04-22T21:01:38.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.830+0000 7f0211ffb700 1 -- 172.21.15.38:0/4289231869 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0218010e70 con 0x7f021c102910 2024-04-22T21:01:38.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.830+0000 7f0223660700 1 -- 172.21.15.38:0/4289231869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021c111e90 con 0x7f021c102910 2024-04-22T21:01:38.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.830+0000 7f0211ffb700 1 -- 172.21.15.38:0/4289231869 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f021801ae70 con 0x7f021c102910 2024-04-22T21:01:38.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.830+0000 7f0223660700 1 -- 172.21.15.38:0/4289231869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f021c112a00 con 0x7f021c102910 2024-04-22T21:01:38.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.831+0000 7f0211ffb700 1 -- 172.21.15.38:0/4289231869 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) v1 ==== 43389+0+0 (secure 0 0 0) 0x7f02180244d0 con 0x7f021c102910 2024-04-22T21:01:38.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.831+0000 7f0223660700 1 -- 172.21.15.38:0/4289231869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f021c05f5f0 con 0x7f021c102910 2024-04-22T21:01:38.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.831+0000 7f0211ffb700 1 --2- 172.21.15.38:0/4289231869 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f02080367e0 0x7f0208038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:38.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.831+0000 7f0211ffb700 1 -- 172.21.15.38:0/4289231869 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f021804adf0 con 0x7f021c102910 2024-04-22T21:01:38.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.832+0000 7f0220bfb700 1 --2- 172.21.15.38:0/4289231869 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f02080367e0 0x7f0208038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:38.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.832+0000 7f0220bfb700 1 --2- 172.21.15.38:0/4289231869 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f02080367e0 0x7f0208038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f020c009990 tx=0x7f020c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:38.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.836+0000 7f0211ffb700 1 -- 172.21.15.38:0/4289231869 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f0218018350 con 0x7f021c102910 2024-04-22T21:01:38.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:37.956+0000 7f0211ffb700 1 -- 172.21.15.38:0/4289231869 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f021801a780 con 0x7f021c102910 2024-04-22T21:01:38.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:38.034+0000 7f0223660700 1 -- 172.21.15.38:0/4289231869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f021c04bed0 con 0x7f021c102910 2024-04-22T21:01:38.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:38.035+0000 7f0211ffb700 1 -- 172.21.15.38:0/4289231869 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f0218018350 con 0x7f021c102910 2024-04-22T21:01:38.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:38.037+0000 7f0223660700 1 -- 172.21.15.38:0/4289231869 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f02080367e0 msgr2=0x7f0208038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:38.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:38.037+0000 7f0223660700 1 --2- 172.21.15.38:0/4289231869 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f02080367e0 0x7f0208038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f020c009990 tx=0x7f020c006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:38.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:38.037+0000 7f0223660700 1 -- 172.21.15.38:0/4289231869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f021c102910 msgr2=0x7f021c111750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:38.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:38.037+0000 7f0223660700 1 --2- 172.21.15.38:0/4289231869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f021c102910 0x7f021c111750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0218006b00 tx=0x7f0218009250 comp rx=0 tx=0).stop 2024-04-22T21:01:38.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:38.037+0000 7f0223660700 1 -- 172.21.15.38:0/4289231869 shutdown_connections 2024-04-22T21:01:38.617 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:38.037+0000 7f0223660700 1 --2- 172.21.15.38:0/4289231869 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f02080367e0 0x7f0208038c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:38.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:38.037+0000 7f0223660700 1 --2- 172.21.15.38:0/4289231869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f021c102910 0x7f021c111750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:38.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:38.037+0000 7f0223660700 1 -- 172.21.15.38:0/4289231869 >> 172.21.15.38:0/4289231869 conn(0x7f021c0fe0b0 msgr2=0x7f021c0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:38.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:38.037+0000 7f0223660700 1 -- 172.21.15.38:0/4289231869 shutdown_connections 2024-04-22T21:01:38.618 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:38.037+0000 7f0223660700 1 -- 172.21.15.38:0/4289231869 wait complete. 2024-04-22T21:01:39.360 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:38 smithi038 ceph-mon[26374]: mgrmap e13: smithi038.lxfgut(active, since 6s) 2024-04-22T21:01:39.920 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.053+0000 7f8523d66700 1 Processor -- start 2024-04-22T21:01:39.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.053+0000 7f8523d66700 1 -- start start 2024-04-22T21:01:39.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.053+0000 7f8523d66700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f851c101d80 0x7f851c102190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:39.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.053+0000 7f8523d66700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851c1026d0 con 0x7f851c101d80 2024-04-22T21:01:39.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.054+0000 7f8521b02700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f851c101d80 0x7f851c102190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:39.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.054+0000 7f8521b02700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f851c101d80 0x7f851c102190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46716/0 (socket says 172.21.15.38:46716) 2024-04-22T21:01:39.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.054+0000 7f8521b02700 1 -- 172.21.15.38:0/4013082690 learned_addr learned my addr 172.21.15.38:0/4013082690 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:39.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.054+0000 7f8521b02700 1 -- 172.21.15.38:0/4013082690 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f851c102810 con 0x7f851c101d80 2024-04-22T21:01:39.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.054+0000 7f8521b02700 1 --2- 172.21.15.38:0/4013082690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f851c101d80 0x7f851c102190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f85180098e0 tx=0x7f8518009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41b923c6bd0a50c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:39.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.054+0000 7f8520b00700 1 -- 172.21.15.38:0/4013082690 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f851800b860 con 0x7f851c101d80 2024-04-22T21:01:39.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.054+0000 7f8520b00700 1 -- 172.21.15.38:0/4013082690 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f851800be90 con 0x7f851c101d80 2024-04-22T21:01:39.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.054+0000 7f8520b00700 1 -- 172.21.15.38:0/4013082690 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85180103c0 con 0x7f851c101d80 2024-04-22T21:01:39.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.055+0000 7f8523d66700 1 -- 172.21.15.38:0/4013082690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f851c101d80 msgr2=0x7f851c102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:39.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.055+0000 7f8523d66700 1 --2- 172.21.15.38:0/4013082690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f851c101d80 0x7f851c102190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f85180098e0 tx=0x7f8518009250 comp rx=0 tx=0).stop 2024-04-22T21:01:39.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.055+0000 7f8523d66700 1 -- 172.21.15.38:0/4013082690 shutdown_connections 2024-04-22T21:01:39.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.055+0000 7f8523d66700 1 --2- 172.21.15.38:0/4013082690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f851c101d80 0x7f851c102190 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:39.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.055+0000 7f8523d66700 1 -- 172.21.15.38:0/4013082690 >> 172.21.15.38:0/4013082690 conn(0x7f851c0fc6b0 msgr2=0x7f851c0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:39.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.055+0000 7f8523d66700 1 -- 172.21.15.38:0/4013082690 shutdown_connections 2024-04-22T21:01:39.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.055+0000 7f8523d66700 1 -- 172.21.15.38:0/4013082690 wait complete. 2024-04-22T21:01:39.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.056+0000 7f8523d66700 1 Processor -- start 2024-04-22T21:01:39.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.056+0000 7f8523d66700 1 -- start start 2024-04-22T21:01:39.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.056+0000 7f8523d66700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f851c101d80 0x7f851c107a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:39.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.056+0000 7f8523d66700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851c107f80 con 0x7f851c101d80 2024-04-22T21:01:39.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.056+0000 7f8521b02700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f851c101d80 0x7f851c107a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:39.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.056+0000 7f8521b02700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f851c101d80 0x7f851c107a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46732/0 (socket says 172.21.15.38:46732) 2024-04-22T21:01:39.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.056+0000 7f8521b02700 1 -- 172.21.15.38:0/3615338962 learned_addr learned my addr 172.21.15.38:0/3615338962 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:39.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.056+0000 7f8521b02700 1 -- 172.21.15.38:0/3615338962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85180094f0 con 0x7f851c101d80 2024-04-22T21:01:39.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.056+0000 7f8521b02700 1 --2- 172.21.15.38:0/3615338962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f851c101d80 0x7f851c107a40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8518011680 tx=0x7f8518009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:39.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.056+0000 7f85127fc700 1 -- 172.21.15.38:0/3615338962 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8518010890 con 0x7f851c101d80 2024-04-22T21:01:39.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.057+0000 7f85127fc700 1 -- 172.21.15.38:0/3615338962 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8518010ec0 con 0x7f851c101d80 2024-04-22T21:01:39.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.057+0000 7f8523d66700 1 -- 172.21.15.38:0/3615338962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f851c105ff0 con 0x7f851c101d80 2024-04-22T21:01:39.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.057+0000 7f85127fc700 1 -- 172.21.15.38:0/3615338962 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85180243f0 con 0x7f851c101d80 2024-04-22T21:01:39.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.057+0000 7f8523d66700 1 -- 172.21.15.38:0/3615338962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f851c1064e0 con 0x7f851c101d80 2024-04-22T21:01:39.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.057+0000 7f85127fc700 1 -- 172.21.15.38:0/3615338962 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f851802d3f0 con 0x7f851c101d80 2024-04-22T21:01:39.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.058+0000 7f8523d66700 1 -- 172.21.15.38:0/3615338962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f851c04bed0 con 0x7f851c101d80 2024-04-22T21:01:39.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.058+0000 7f85127fc700 1 --2- 172.21.15.38:0/3615338962 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f8508036880 0x7f8508038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:39.924 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.058+0000 7f85127fc700 1 -- 172.21.15.38:0/3615338962 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f851804a490 con 0x7f851c101d80 2024-04-22T21:01:39.924 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.058+0000 7f8521301700 1 --2- 172.21.15.38:0/3615338962 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f8508036880 0x7f8508038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:39.924 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.058+0000 7f8521301700 1 --2- 172.21.15.38:0/3615338962 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f8508036880 0x7f8508038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f850c009990 tx=0x7f850c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:39.924 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.062+0000 7f85127fc700 1 -- 172.21.15.38:0/3615338962 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f851802b8e0 con 0x7f851c101d80 2024-04-22T21:01:39.924 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.364+0000 7f8523d66700 1 -- 172.21.15.38:0/3615338962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f851c103650 con 0x7f851c101d80 2024-04-22T21:01:39.924 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.371+0000 7f85127fc700 1 -- 172.21.15.38:0/3615338962 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f8518006d70 con 0x7f851c101d80 2024-04-22T21:01:39.924 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.373+0000 7f8523d66700 1 -- 172.21.15.38:0/3615338962 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f8508036880 msgr2=0x7f8508038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:39.924 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.373+0000 7f8523d66700 1 --2- 172.21.15.38:0/3615338962 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f8508036880 0x7f8508038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f850c009990 tx=0x7f850c006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:39.924 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.373+0000 7f8523d66700 1 -- 172.21.15.38:0/3615338962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f851c101d80 msgr2=0x7f851c107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:39.924 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.373+0000 7f8523d66700 1 --2- 172.21.15.38:0/3615338962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f851c101d80 0x7f851c107a40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8518011680 tx=0x7f8518009250 comp rx=0 tx=0).stop 2024-04-22T21:01:39.924 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.373+0000 7f8523d66700 1 -- 172.21.15.38:0/3615338962 shutdown_connections 2024-04-22T21:01:39.925 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.373+0000 7f8523d66700 1 --2- 172.21.15.38:0/3615338962 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f8508036880 0x7f8508038d30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:39.925 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.373+0000 7f8523d66700 1 --2- 172.21.15.38:0/3615338962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f851c101d80 0x7f851c107a40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:39.925 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.373+0000 7f8523d66700 1 -- 172.21.15.38:0/3615338962 >> 172.21.15.38:0/3615338962 conn(0x7f851c0fc6b0 msgr2=0x7f851c0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:39.925 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.373+0000 7f8523d66700 1 -- 172.21.15.38:0/3615338962 shutdown_connections 2024-04-22T21:01:39.925 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T21:01:39.373+0000 7f8523d66700 1 -- 172.21.15.38:0/3615338962 wait complete. 2024-04-22T21:01:39.925 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-22T21:01:39.925 INFO:teuthology.orchestra.run.smithi038.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-22T21:01:39.925 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:01:39.925 INFO:teuthology.orchestra.run.smithi038.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:01:39.925 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:01:39.925 INFO:teuthology.orchestra.run.smithi038.stdout:Or, if you are only running a single cluster on this host: 2024-04-22T21:01:39.925 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:01:39.926 INFO:teuthology.orchestra.run.smithi038.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-22T21:01:39.926 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:01:39.926 INFO:teuthology.orchestra.run.smithi038.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-22T21:01:39.926 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:01:39.926 INFO:teuthology.orchestra.run.smithi038.stdout: ceph telemetry on 2024-04-22T21:01:39.926 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:01:39.926 INFO:teuthology.orchestra.run.smithi038.stdout:For more information see: 2024-04-22T21:01:39.926 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:01:39.926 INFO:teuthology.orchestra.run.smithi038.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-22T21:01:39.926 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:01:39.926 INFO:teuthology.orchestra.run.smithi038.stdout:Bootstrap complete. 2024-04-22T21:01:39.960 INFO:tasks.cephadm:Fetching config... 2024-04-22T21:01:39.961 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T21:01:39.961 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-22T21:01:40.018 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-22T21:01:40.019 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T21:01:40.019 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-22T21:01:40.075 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-22T21:01:40.075 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T21:01:40.075 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/keyring of=/dev/stdout 2024-04-22T21:01:40.140 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-22T21:01:40.140 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T21:01:40.140 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-22T21:01:40.195 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-22T21:01:40.195 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwKQaHt7Hq/TAdPcNj15MTJBeTmwjU51OPBip51SAd/PK7M7x8PwFUVGlWm+dh/qoSbqu5uTYyKv326j4g1ithmDBuYUH6tFgh24ICByEpcPVppSJRZ7ZLgU6M4IlDZaM9B3AEjjjyIbtM1Ij3e2r0OGLyvG3Gvvrys1GeYygCl5k41LiBI/8QTmClGQF5F/rb06ExAt1ZgIrAzJeNoDynx6tZpVtScTE8xmi56Gt+f4SE3cSooxLPUS2ksFv/08fTZY5JvkS0QA/A9iC6fda0OFWNn9+nzdK7CPOm8L+Tyojh9WnytzQzbAS/BBfsQZHs6y9Hv7BD1n4CNk/6yquq/8xKu4FnZ+gnU4kL8sBRnTOB7YieIQ6jFTPtBo0Im5PFAQYK3X/SjS1SudYXt53wMuv/cNRnMXsCxHLSo6k+6svzxgbTlb7dSs2C2caNHLkwVU/7Gc1pJqSZ7lLnG8CxQniOjiyc+eq1ZGnOquYM1F4vjfGvJ5j+Rp27bEJDY2k= ceph-3c5f5352-00eb-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T21:01:40.274 INFO:teuthology.orchestra.run.smithi038.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwKQaHt7Hq/TAdPcNj15MTJBeTmwjU51OPBip51SAd/PK7M7x8PwFUVGlWm+dh/qoSbqu5uTYyKv326j4g1ithmDBuYUH6tFgh24ICByEpcPVppSJRZ7ZLgU6M4IlDZaM9B3AEjjjyIbtM1Ij3e2r0OGLyvG3Gvvrys1GeYygCl5k41LiBI/8QTmClGQF5F/rb06ExAt1ZgIrAzJeNoDynx6tZpVtScTE8xmi56Gt+f4SE3cSooxLPUS2ksFv/08fTZY5JvkS0QA/A9iC6fda0OFWNn9+nzdK7CPOm8L+Tyojh9WnytzQzbAS/BBfsQZHs6y9Hv7BD1n4CNk/6yquq/8xKu4FnZ+gnU4kL8sBRnTOB7YieIQ6jFTPtBo0Im5PFAQYK3X/SjS1SudYXt53wMuv/cNRnMXsCxHLSo6k+6svzxgbTlb7dSs2C2caNHLkwVU/7Gc1pJqSZ7lLnG8CxQniOjiyc+eq1ZGnOquYM1F4vjfGvJ5j+Rp27bEJDY2k= ceph-3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T21:01:40.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwKQaHt7Hq/TAdPcNj15MTJBeTmwjU51OPBip51SAd/PK7M7x8PwFUVGlWm+dh/qoSbqu5uTYyKv326j4g1ithmDBuYUH6tFgh24ICByEpcPVppSJRZ7ZLgU6M4IlDZaM9B3AEjjjyIbtM1Ij3e2r0OGLyvG3Gvvrys1GeYygCl5k41LiBI/8QTmClGQF5F/rb06ExAt1ZgIrAzJeNoDynx6tZpVtScTE8xmi56Gt+f4SE3cSooxLPUS2ksFv/08fTZY5JvkS0QA/A9iC6fda0OFWNn9+nzdK7CPOm8L+Tyojh9WnytzQzbAS/BBfsQZHs6y9Hv7BD1n4CNk/6yquq/8xKu4FnZ+gnU4kL8sBRnTOB7YieIQ6jFTPtBo0Im5PFAQYK3X/SjS1SudYXt53wMuv/cNRnMXsCxHLSo6k+6svzxgbTlb7dSs2C2caNHLkwVU/7Gc1pJqSZ7lLnG8CxQniOjiyc+eq1ZGnOquYM1F4vjfGvJ5j+Rp27bEJDY2k= ceph-3c5f5352-00eb-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T21:01:40.325 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwKQaHt7Hq/TAdPcNj15MTJBeTmwjU51OPBip51SAd/PK7M7x8PwFUVGlWm+dh/qoSbqu5uTYyKv326j4g1ithmDBuYUH6tFgh24ICByEpcPVppSJRZ7ZLgU6M4IlDZaM9B3AEjjjyIbtM1Ij3e2r0OGLyvG3Gvvrys1GeYygCl5k41LiBI/8QTmClGQF5F/rb06ExAt1ZgIrAzJeNoDynx6tZpVtScTE8xmi56Gt+f4SE3cSooxLPUS2ksFv/08fTZY5JvkS0QA/A9iC6fda0OFWNn9+nzdK7CPOm8L+Tyojh9WnytzQzbAS/BBfsQZHs6y9Hv7BD1n4CNk/6yquq/8xKu4FnZ+gnU4kL8sBRnTOB7YieIQ6jFTPtBo0Im5PFAQYK3X/SjS1SudYXt53wMuv/cNRnMXsCxHLSo6k+6svzxgbTlb7dSs2C2caNHLkwVU/7Gc1pJqSZ7lLnG8CxQniOjiyc+eq1ZGnOquYM1F4vjfGvJ5j+Rp27bEJDY2k= ceph-3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T21:01:40.337 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-22T21:01:40.614 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:01:40.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:40 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3615338962' entity='client.admin' 2024-04-22T21:01:43.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.091+0000 7f0e9b300700 1 -- 172.21.15.38:0/2866513215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e94071e40 msgr2=0x7f0e94070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:43.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.091+0000 7f0e9b300700 1 --2- 172.21.15.38:0/2866513215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e94071e40 0x7f0e94070350 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0e900048c0 tx=0x7f0e90009e30 comp rx=0 tx=0).stop 2024-04-22T21:01:43.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.091+0000 7f0e9b300700 1 -- 172.21.15.38:0/2866513215 shutdown_connections 2024-04-22T21:01:43.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.091+0000 7f0e9b300700 1 --2- 172.21.15.38:0/2866513215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e94071e40 0x7f0e94070350 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:43.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.091+0000 7f0e9b300700 1 -- 172.21.15.38:0/2866513215 >> 172.21.15.38:0/2866513215 conn(0x7f0e940f81c0 msgr2=0x7f0e940fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:43.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.091+0000 7f0e9b300700 1 -- 172.21.15.38:0/2866513215 shutdown_connections 2024-04-22T21:01:43.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.091+0000 7f0e9b300700 1 -- 172.21.15.38:0/2866513215 wait complete. 2024-04-22T21:01:43.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.091+0000 7f0e9b300700 1 Processor -- start 2024-04-22T21:01:43.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.091+0000 7f0e9b300700 1 -- start start 2024-04-22T21:01:43.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.092+0000 7f0e9b300700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e94071e40 0x7f0e9410f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:43.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.092+0000 7f0e9b300700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9410f850 con 0x7f0e94071e40 2024-04-22T21:01:43.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.092+0000 7f0e9909c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e94071e40 0x7f0e9410f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:43.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.092+0000 7f0e9909c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e94071e40 0x7f0e9410f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49514/0 (socket says 172.21.15.38:49514) 2024-04-22T21:01:43.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.092+0000 7f0e9909c700 1 -- 172.21.15.38:0/181971346 learned_addr learned my addr 172.21.15.38:0/181971346 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:43.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.092+0000 7f0e9909c700 1 -- 172.21.15.38:0/181971346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e90005040 con 0x7f0e94071e40 2024-04-22T21:01:43.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.092+0000 7f0e9909c700 1 --2- 172.21.15.38:0/181971346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e94071e40 0x7f0e9410f310 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009680 tx=0x7f0e90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:43.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.092+0000 7f0e89ffb700 1 -- 172.21.15.38:0/181971346 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e9000bed0 con 0x7f0e94071e40 2024-04-22T21:01:43.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.092+0000 7f0e89ffb700 1 -- 172.21.15.38:0/181971346 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e9000b5c0 con 0x7f0e94071e40 2024-04-22T21:01:43.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.092+0000 7f0e9b300700 1 -- 172.21.15.38:0/181971346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e9410fa50 con 0x7f0e94071e40 2024-04-22T21:01:43.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.092+0000 7f0e89ffb700 1 -- 172.21.15.38:0/181971346 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e9001c5b0 con 0x7f0e94071e40 2024-04-22T21:01:43.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.093+0000 7f0e9b300700 1 -- 172.21.15.38:0/181971346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e941105c0 con 0x7f0e94071e40 2024-04-22T21:01:43.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.093+0000 7f0e89ffb700 1 -- 172.21.15.38:0/181971346 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f0e90013420 con 0x7f0e94071e40 2024-04-22T21:01:43.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.093+0000 7f0e9b300700 1 -- 172.21.15.38:0/181971346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e94105180 con 0x7f0e94071e40 2024-04-22T21:01:43.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.094+0000 7f0e89ffb700 1 --2- 172.21.15.38:0/181971346 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f0e80036880 0x7f0e80038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:43.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.094+0000 7f0e89ffb700 1 -- 172.21.15.38:0/181971346 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0e90049980 con 0x7f0e94071e40 2024-04-22T21:01:43.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.094+0000 7f0e9889b700 1 --2- 172.21.15.38:0/181971346 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f0e80036880 0x7f0e80038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:43.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.094+0000 7f0e9889b700 1 --2- 172.21.15.38:0/181971346 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f0e80036880 0x7f0e80038d30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0e84009990 tx=0x7f0e84006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:43.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.098+0000 7f0e89ffb700 1 -- 172.21.15.38:0/181971346 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f0e90013a80 con 0x7f0e94071e40 2024-04-22T21:01:43.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.297+0000 7f0e9b300700 1 -- 172.21.15.38:0/181971346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f0e9405f5f0 con 0x7f0e94071e40 2024-04-22T21:01:43.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.301+0000 7f0e89ffb700 1 -- 172.21.15.38:0/181971346 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0e90011370 con 0x7f0e94071e40 2024-04-22T21:01:43.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.303+0000 7f0e9b300700 1 -- 172.21.15.38:0/181971346 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f0e80036880 msgr2=0x7f0e80038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:43.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.303+0000 7f0e9b300700 1 --2- 172.21.15.38:0/181971346 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f0e80036880 0x7f0e80038d30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0e84009990 tx=0x7f0e84006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:43.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.303+0000 7f0e9b300700 1 -- 172.21.15.38:0/181971346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e94071e40 msgr2=0x7f0e9410f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:43.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.303+0000 7f0e9b300700 1 --2- 172.21.15.38:0/181971346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e94071e40 0x7f0e9410f310 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009680 tx=0x7f0e90009e30 comp rx=0 tx=0).stop 2024-04-22T21:01:43.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.303+0000 7f0e9b300700 1 -- 172.21.15.38:0/181971346 shutdown_connections 2024-04-22T21:01:43.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.303+0000 7f0e9b300700 1 --2- 172.21.15.38:0/181971346 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f0e80036880 0x7f0e80038d30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:43.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.303+0000 7f0e9b300700 1 --2- 172.21.15.38:0/181971346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e94071e40 0x7f0e9410f310 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:43.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.303+0000 7f0e9b300700 1 -- 172.21.15.38:0/181971346 >> 172.21.15.38:0/181971346 conn(0x7f0e940f81c0 msgr2=0x7f0e940f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:43.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.304+0000 7f0e9b300700 1 -- 172.21.15.38:0/181971346 shutdown_connections 2024-04-22T21:01:43.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:43.304+0000 7f0e9b300700 1 -- 172.21.15.38:0/181971346 wait complete. 2024-04-22T21:01:44.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:44 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/181971346' entity='client.admin' 2024-04-22T21:01:44.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:44 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:44.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:44 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:44.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:44 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:01:44.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:44 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:44.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:44 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi038", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T21:01:44.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:44 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi038", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-22T21:01:44.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:44 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:01:44.600 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-22T21:01:44.601 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-22T21:01:45.304 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:01:45.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:45 smithi038 ceph-mon[26374]: Deploying daemon crash.smithi038 on smithi038 2024-04-22T21:01:47.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.452+0000 7f6610803700 1 -- 172.21.15.38:0/1717280007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f660c0fb7d0 msgr2=0x7f660c0fbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:47.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.452+0000 7f6610803700 1 --2- 172.21.15.38:0/1717280007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f660c0fb7d0 0x7f660c0fbbe0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f65fc0048c0 tx=0x7f65fc009e30 comp rx=0 tx=0).stop 2024-04-22T21:01:47.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.452+0000 7f6610803700 1 -- 172.21.15.38:0/1717280007 shutdown_connections 2024-04-22T21:01:47.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.452+0000 7f6610803700 1 --2- 172.21.15.38:0/1717280007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f660c0fb7d0 0x7f660c0fbbe0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:47.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.452+0000 7f6610803700 1 -- 172.21.15.38:0/1717280007 >> 172.21.15.38:0/1717280007 conn(0x7f660c0f6f30 msgr2=0x7f660c0f9360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:47.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.452+0000 7f6610803700 1 -- 172.21.15.38:0/1717280007 shutdown_connections 2024-04-22T21:01:47.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.452+0000 7f6610803700 1 -- 172.21.15.38:0/1717280007 wait complete. 2024-04-22T21:01:47.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.453+0000 7f6610803700 1 Processor -- start 2024-04-22T21:01:47.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.453+0000 7f6610803700 1 -- start start 2024-04-22T21:01:47.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.453+0000 7f6610803700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f660c0fb7d0 0x7f660c10cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:47.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.453+0000 7f6610803700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660c10d420 con 0x7f660c0fb7d0 2024-04-22T21:01:47.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.453+0000 7f660ad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f660c0fb7d0 0x7f660c10cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.453+0000 7f660ad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f660c0fb7d0 0x7f660c10cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49540/0 (socket says 172.21.15.38:49540) 2024-04-22T21:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.453+0000 7f660ad9d700 1 -- 172.21.15.38:0/3123927256 learned_addr learned my addr 172.21.15.38:0/3123927256 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.454+0000 7f660ad9d700 1 -- 172.21.15.38:0/3123927256 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65fc005040 con 0x7f660c0fb7d0 2024-04-22T21:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.454+0000 7f660ad9d700 1 --2- 172.21.15.38:0/3123927256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f660c0fb7d0 0x7f660c10cee0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f65fc0096b0 tx=0x7f65fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.454+0000 7f65f37fe700 1 -- 172.21.15.38:0/3123927256 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65fc00bed0 con 0x7f660c0fb7d0 2024-04-22T21:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.454+0000 7f65f37fe700 1 -- 172.21.15.38:0/3123927256 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f65fc00b5c0 con 0x7f660c0fb7d0 2024-04-22T21:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.454+0000 7f6610803700 1 -- 172.21.15.38:0/3123927256 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f660c10d620 con 0x7f660c0fb7d0 2024-04-22T21:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.454+0000 7f65f37fe700 1 -- 172.21.15.38:0/3123927256 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65fc01c5b0 con 0x7f660c0fb7d0 2024-04-22T21:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.454+0000 7f6610803700 1 -- 172.21.15.38:0/3123927256 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f660c10e190 con 0x7f660c0fb7d0 2024-04-22T21:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.455+0000 7f65f37fe700 1 -- 172.21.15.38:0/3123927256 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f65fc01c710 con 0x7f660c0fb7d0 2024-04-22T21:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.455+0000 7f65f37fe700 1 --2- 172.21.15.38:0/3123927256 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f65f40368d0 0x7f65f4038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.455+0000 7f6610803700 1 -- 172.21.15.38:0/3123927256 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f660c101ce0 con 0x7f660c0fb7d0 2024-04-22T21:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.455+0000 7f65f37fe700 1 -- 172.21.15.38:0/3123927256 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f65fc049ae0 con 0x7f660c0fb7d0 2024-04-22T21:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.455+0000 7f660a59c700 1 --2- 172.21.15.38:0/3123927256 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f65f40368d0 0x7f65f4038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.456+0000 7f660a59c700 1 --2- 172.21.15.38:0/3123927256 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f65f40368d0 0x7f65f4038d80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6600009990 tx=0x7f6600006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:47.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.460+0000 7f65f37fe700 1 -- 172.21.15.38:0/3123927256 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f65fc025400 con 0x7f660c0fb7d0 2024-04-22T21:01:47.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.663+0000 7f6610803700 1 -- 172.21.15.38:0/3123927256 --> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f660c066200 con 0x7f65f40368d0 2024-04-22T21:01:47.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.667+0000 7f65f37fe700 1 -- 172.21.15.38:0/3123927256 <== mgr.14162 v2:172.21.15.38:6800/989134174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f660c066200 con 0x7f65f40368d0 2024-04-22T21:01:47.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.669+0000 7f6610803700 1 -- 172.21.15.38:0/3123927256 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f65f40368d0 msgr2=0x7f65f4038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:47.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.669+0000 7f6610803700 1 --2- 172.21.15.38:0/3123927256 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f65f40368d0 0x7f65f4038d80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6600009990 tx=0x7f6600006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:47.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.669+0000 7f6610803700 1 -- 172.21.15.38:0/3123927256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f660c0fb7d0 msgr2=0x7f660c10cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:47.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.669+0000 7f6610803700 1 --2- 172.21.15.38:0/3123927256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f660c0fb7d0 0x7f660c10cee0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f65fc0096b0 tx=0x7f65fc009e30 comp rx=0 tx=0).stop 2024-04-22T21:01:47.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.669+0000 7f6610803700 1 -- 172.21.15.38:0/3123927256 shutdown_connections 2024-04-22T21:01:47.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.669+0000 7f6610803700 1 --2- 172.21.15.38:0/3123927256 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f65f40368d0 0x7f65f4038d80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:47.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.669+0000 7f6610803700 1 --2- 172.21.15.38:0/3123927256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f660c0fb7d0 0x7f660c10cee0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:47.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.669+0000 7f6610803700 1 -- 172.21.15.38:0/3123927256 >> 172.21.15.38:0/3123927256 conn(0x7f660c0f6f30 msgr2=0x7f660c065af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:47.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.669+0000 7f6610803700 1 -- 172.21.15.38:0/3123927256 shutdown_connections 2024-04-22T21:01:47.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:47.669+0000 7f6610803700 1 -- 172.21.15.38:0/3123927256 wait complete. 2024-04-22T21:01:48.259 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi098 2024-04-22T21:01:48.259 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T21:01:48.259 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/etc/ceph/ceph.conf 2024-04-22T21:01:48.278 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T21:01:48.278 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:01:48.336 INFO:tasks.cephadm:Adding host smithi098 to orchestrator... 2024-04-22T21:01:48.336 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph orch host add smithi098 2024-04-22T21:01:48.617 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:01:48.766 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:48 smithi038 ceph-mon[26374]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:01:48.766 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:48 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:48.766 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:48 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:48.766 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:48 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:48.767 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:48 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:48.767 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:48 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:49.806 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:49 smithi038 ceph-mon[26374]: Deploying daemon node-exporter.smithi038 on smithi038 2024-04-22T21:01:50.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.326+0000 7fda99337700 1 -- 172.21.15.38:0/825472731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda940fca60 msgr2=0x7fda940fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:50.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.326+0000 7fda99337700 1 --2- 172.21.15.38:0/825472731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda940fca60 0x7fda940fce70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fda840048c0 tx=0x7fda84009e30 comp rx=0 tx=0).stop 2024-04-22T21:01:50.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.326+0000 7fda99337700 1 -- 172.21.15.38:0/825472731 shutdown_connections 2024-04-22T21:01:50.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.326+0000 7fda99337700 1 --2- 172.21.15.38:0/825472731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda940fca60 0x7fda940fce70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:50.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.326+0000 7fda99337700 1 -- 172.21.15.38:0/825472731 >> 172.21.15.38:0/825472731 conn(0x7fda940f81c0 msgr2=0x7fda940fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:50.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.326+0000 7fda99337700 1 -- 172.21.15.38:0/825472731 shutdown_connections 2024-04-22T21:01:50.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.326+0000 7fda99337700 1 -- 172.21.15.38:0/825472731 wait complete. 2024-04-22T21:01:50.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.327+0000 7fda99337700 1 Processor -- start 2024-04-22T21:01:50.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.327+0000 7fda99337700 1 -- start start 2024-04-22T21:01:50.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.327+0000 7fda99337700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda940fca60 0x7fda9410d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:50.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.327+0000 7fda99337700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9410d5f0 con 0x7fda940fca60 2024-04-22T21:01:50.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.327+0000 7fda92ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda940fca60 0x7fda9410d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:50.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.327+0000 7fda92ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda940fca60 0x7fda9410d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49582/0 (socket says 172.21.15.38:49582) 2024-04-22T21:01:50.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.327+0000 7fda92ffd700 1 -- 172.21.15.38:0/2870835727 learned_addr learned my addr 172.21.15.38:0/2870835727 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:50.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.327+0000 7fda92ffd700 1 -- 172.21.15.38:0/2870835727 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda84005040 con 0x7fda940fca60 2024-04-22T21:01:50.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.328+0000 7fda92ffd700 1 --2- 172.21.15.38:0/2870835727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda940fca60 0x7fda9410d0b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fda840096b0 tx=0x7fda84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:50.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.328+0000 7fda7b7fe700 1 -- 172.21.15.38:0/2870835727 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda8400bed0 con 0x7fda940fca60 2024-04-22T21:01:50.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.328+0000 7fda7b7fe700 1 -- 172.21.15.38:0/2870835727 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda8400b5c0 con 0x7fda940fca60 2024-04-22T21:01:50.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.328+0000 7fda99337700 1 -- 172.21.15.38:0/2870835727 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda9410d7f0 con 0x7fda940fca60 2024-04-22T21:01:50.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.328+0000 7fda7b7fe700 1 -- 172.21.15.38:0/2870835727 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda8401c5b0 con 0x7fda940fca60 2024-04-22T21:01:50.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.328+0000 7fda99337700 1 -- 172.21.15.38:0/2870835727 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda9410e360 con 0x7fda940fca60 2024-04-22T21:01:50.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.329+0000 7fda7b7fe700 1 -- 172.21.15.38:0/2870835727 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fda8401c820 con 0x7fda940fca60 2024-04-22T21:01:50.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.329+0000 7fda99337700 1 -- 172.21.15.38:0/2870835727 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda94102f70 con 0x7fda940fca60 2024-04-22T21:01:50.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.329+0000 7fda7b7fe700 1 --2- 172.21.15.38:0/2870835727 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fda7c036880 0x7fda7c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:50.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.329+0000 7fda7b7fe700 1 -- 172.21.15.38:0/2870835727 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fda84049cf0 con 0x7fda940fca60 2024-04-22T21:01:50.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.329+0000 7fda927fc700 1 --2- 172.21.15.38:0/2870835727 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fda7c036880 0x7fda7c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:50.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.329+0000 7fda927fc700 1 --2- 172.21.15.38:0/2870835727 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fda7c036880 0x7fda7c038d30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fda88009990 tx=0x7fda88006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:50.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.334+0000 7fda7b7fe700 1 -- 172.21.15.38:0/2870835727 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fda84023300 con 0x7fda940fca60 2024-04-22T21:01:50.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:50.533+0000 7fda99337700 1 -- 172.21.15.38:0/2870835727 --> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi098", "target": ["mon-mgr", ""]}) v1 -- 0x7fda9410e5a0 con 0x7fda7c036880 2024-04-22T21:01:52.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:52 smithi038 ceph-mon[26374]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:01:52.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:52 smithi038 ceph-mon[26374]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:01:52.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:52 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:52.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:52 smithi038 ceph-mon[26374]: Deploying cephadm binary to smithi098 2024-04-22T21:01:52.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:52.417+0000 7fda7b7fe700 1 -- 172.21.15.38:0/2870835727 <== mgr.14162 v2:172.21.15.38:6800/989134174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fda9410e5a0 con 0x7fda7c036880 2024-04-22T21:01:52.413 INFO:teuthology.orchestra.run.smithi038.stdout:Added host 'smithi098' with addr '172.21.15.98' 2024-04-22T21:01:52.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:52.419+0000 7fda99337700 1 -- 172.21.15.38:0/2870835727 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fda7c036880 msgr2=0x7fda7c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:52.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:52.419+0000 7fda99337700 1 --2- 172.21.15.38:0/2870835727 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fda7c036880 0x7fda7c038d30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fda88009990 tx=0x7fda88006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:52.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:52.419+0000 7fda99337700 1 -- 172.21.15.38:0/2870835727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda940fca60 msgr2=0x7fda9410d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:52.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:52.419+0000 7fda99337700 1 --2- 172.21.15.38:0/2870835727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda940fca60 0x7fda9410d0b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fda840096b0 tx=0x7fda84009e30 comp rx=0 tx=0).stop 2024-04-22T21:01:52.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:52.419+0000 7fda99337700 1 -- 172.21.15.38:0/2870835727 shutdown_connections 2024-04-22T21:01:52.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:52.419+0000 7fda99337700 1 --2- 172.21.15.38:0/2870835727 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fda7c036880 0x7fda7c038d30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:52.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:52.419+0000 7fda99337700 1 --2- 172.21.15.38:0/2870835727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda940fca60 0x7fda9410d0b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:52.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:52.419+0000 7fda99337700 1 -- 172.21.15.38:0/2870835727 >> 172.21.15.38:0/2870835727 conn(0x7fda940f81c0 msgr2=0x7fda94103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:52.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:52.419+0000 7fda99337700 1 -- 172.21.15.38:0/2870835727 shutdown_connections 2024-04-22T21:01:52.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:52.419+0000 7fda99337700 1 -- 172.21.15.38:0/2870835727 wait complete. 2024-04-22T21:01:53.446 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T21:01:53.657 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:53 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:53.657 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:53 smithi038 ceph-mon[26374]: Added host smithi098 2024-04-22T21:01:53.657 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:53 smithi038 ceph-mon[26374]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:01:53.894 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:01:55.057 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:54 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:55.057 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:54 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:55.058 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:54 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:55.058 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:54 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:55.058 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:54 smithi038 ceph-mon[26374]: Deploying daemon alertmanager.smithi038 on smithi038 2024-04-22T21:01:55.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.558+0000 7f3320ba3700 1 -- 172.21.15.38:0/1129361765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f331c0fca60 msgr2=0x7f331c0fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:55.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.558+0000 7f3320ba3700 1 --2- 172.21.15.38:0/1129361765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f331c0fca60 0x7f331c0fce70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3314009c60 tx=0x7f3314004d10 comp rx=0 tx=0).stop 2024-04-22T21:01:55.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.558+0000 7f3320ba3700 1 -- 172.21.15.38:0/1129361765 shutdown_connections 2024-04-22T21:01:55.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.558+0000 7f3320ba3700 1 --2- 172.21.15.38:0/1129361765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f331c0fca60 0x7f331c0fce70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:55.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.558+0000 7f3320ba3700 1 -- 172.21.15.38:0/1129361765 >> 172.21.15.38:0/1129361765 conn(0x7f331c0f81c0 msgr2=0x7f331c0fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:55.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.558+0000 7f3320ba3700 1 -- 172.21.15.38:0/1129361765 shutdown_connections 2024-04-22T21:01:55.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.558+0000 7f3320ba3700 1 -- 172.21.15.38:0/1129361765 wait complete. 2024-04-22T21:01:55.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.558+0000 7f3320ba3700 1 Processor -- start 2024-04-22T21:01:55.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.558+0000 7f3320ba3700 1 -- start start 2024-04-22T21:01:55.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.559+0000 7f3320ba3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f331c0fca60 0x7f331c071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:55.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.559+0000 7f3320ba3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331c0722a0 con 0x7f331c0fca60 2024-04-22T21:01:55.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.559+0000 7f331a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f331c0fca60 0x7f331c071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:55.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.559+0000 7f331a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f331c0fca60 0x7f331c071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53612/0 (socket says 172.21.15.38:53612) 2024-04-22T21:01:55.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.559+0000 7f331a59c700 1 -- 172.21.15.38:0/3354307360 learned_addr learned my addr 172.21.15.38:0/3354307360 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:55.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.559+0000 7f331a59c700 1 -- 172.21.15.38:0/3354307360 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3314009940 con 0x7f331c0fca60 2024-04-22T21:01:55.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.559+0000 7f331a59c700 1 --2- 172.21.15.38:0/3354307360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f331c0fca60 0x7f331c071d60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3314004e60 tx=0x7f3314004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:55.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.559+0000 7f3312ffd700 1 -- 172.21.15.38:0/3354307360 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3314012410 con 0x7f331c0fca60 2024-04-22T21:01:55.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.559+0000 7f3312ffd700 1 -- 172.21.15.38:0/3354307360 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3314012a40 con 0x7f331c0fca60 2024-04-22T21:01:55.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.559+0000 7f3320ba3700 1 -- 172.21.15.38:0/3354307360 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331c070370 con 0x7f331c0fca60 2024-04-22T21:01:55.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.559+0000 7f3312ffd700 1 -- 172.21.15.38:0/3354307360 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f331401ba40 con 0x7f331c0fca60 2024-04-22T21:01:55.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.559+0000 7f3320ba3700 1 -- 172.21.15.38:0/3354307360 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331c070860 con 0x7f331c0fca60 2024-04-22T21:01:55.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.560+0000 7f3312ffd700 1 -- 172.21.15.38:0/3354307360 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f3314019040 con 0x7f331c0fca60 2024-04-22T21:01:55.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.560+0000 7f3320ba3700 1 -- 172.21.15.38:0/3354307360 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f331c05f5f0 con 0x7f331c0fca60 2024-04-22T21:01:55.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.560+0000 7f3312ffd700 1 --2- 172.21.15.38:0/3354307360 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f33080368d0 0x7f3308038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:55.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.561+0000 7f3312ffd700 1 -- 172.21.15.38:0/3354307360 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3314049c10 con 0x7f331c0fca60 2024-04-22T21:01:55.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.561+0000 7f3319d9b700 1 --2- 172.21.15.38:0/3354307360 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f33080368d0 0x7f3308038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:55.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.561+0000 7f3319d9b700 1 --2- 172.21.15.38:0/3354307360 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f33080368d0 0x7f3308038d80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3304009990 tx=0x7f3304006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:55.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.565+0000 7f3312ffd700 1 -- 172.21.15.38:0/3354307360 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f331402b8d0 con 0x7f331c0fca60 2024-04-22T21:01:55.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.768+0000 7f3320ba3700 1 -- 172.21.15.38:0/3354307360 --> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f331c1064b0 con 0x7f33080368d0 2024-04-22T21:01:55.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.769+0000 7f3312ffd700 1 -- 172.21.15.38:0/3354307360 <== mgr.14162 v2:172.21.15.38:6800/989134174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f331c1064b0 con 0x7f33080368d0 2024-04-22T21:01:55.765 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:01:55.766 INFO:teuthology.orchestra.run.smithi038.stdout:[{"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}] 2024-04-22T21:01:55.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.771+0000 7f3320ba3700 1 -- 172.21.15.38:0/3354307360 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f33080368d0 msgr2=0x7f3308038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:55.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.771+0000 7f3320ba3700 1 --2- 172.21.15.38:0/3354307360 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f33080368d0 0x7f3308038d80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3304009990 tx=0x7f3304006e40 comp rx=0 tx=0).stop 2024-04-22T21:01:55.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.771+0000 7f3320ba3700 1 -- 172.21.15.38:0/3354307360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f331c0fca60 msgr2=0x7f331c071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:55.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.771+0000 7f3320ba3700 1 --2- 172.21.15.38:0/3354307360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f331c0fca60 0x7f331c071d60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3314004e60 tx=0x7f3314004d10 comp rx=0 tx=0).stop 2024-04-22T21:01:55.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.771+0000 7f3320ba3700 1 -- 172.21.15.38:0/3354307360 shutdown_connections 2024-04-22T21:01:55.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.771+0000 7f3320ba3700 1 --2- 172.21.15.38:0/3354307360 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f33080368d0 0x7f3308038d80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:55.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.771+0000 7f3320ba3700 1 --2- 172.21.15.38:0/3354307360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f331c0fca60 0x7f331c071d60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:55.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.771+0000 7f3320ba3700 1 -- 172.21.15.38:0/3354307360 >> 172.21.15.38:0/3354307360 conn(0x7f331c0f81c0 msgr2=0x7f331c103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:55.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.771+0000 7f3320ba3700 1 -- 172.21.15.38:0/3354307360 shutdown_connections 2024-04-22T21:01:55.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:55.771+0000 7f3320ba3700 1 -- 172.21.15.38:0/3354307360 wait complete. 2024-04-22T21:01:56.124 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:55 smithi038 ceph-mon[26374]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:01:56.657 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-22T21:01:56.657 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-22T21:01:56.922 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:01:57.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:57 smithi038 ceph-mon[26374]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:01:57.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:57 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:01:58.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:01:58 smithi038 ceph-mon[26374]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:01:59.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.776+0000 7f6954360700 1 -- 172.21.15.38:0/3058723974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f694c065ce0 msgr2=0x7f694c101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:01:59.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.776+0000 7f6954360700 1 --2- 172.21.15.38:0/3058723974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f694c065ce0 0x7f694c101ae0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6944009c60 tx=0x7f6944004d10 comp rx=0 tx=0).stop 2024-04-22T21:01:59.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.776+0000 7f6954360700 1 -- 172.21.15.38:0/3058723974 shutdown_connections 2024-04-22T21:01:59.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.776+0000 7f6954360700 1 --2- 172.21.15.38:0/3058723974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f694c065ce0 0x7f694c101ae0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:01:59.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.776+0000 7f6954360700 1 -- 172.21.15.38:0/3058723974 >> 172.21.15.38:0/3058723974 conn(0x7f694c0f70d0 msgr2=0x7f694c0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:01:59.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.776+0000 7f6954360700 1 -- 172.21.15.38:0/3058723974 shutdown_connections 2024-04-22T21:01:59.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.776+0000 7f6954360700 1 -- 172.21.15.38:0/3058723974 wait complete. 2024-04-22T21:01:59.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.777+0000 7f6954360700 1 Processor -- start 2024-04-22T21:01:59.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.777+0000 7f6954360700 1 -- start start 2024-04-22T21:01:59.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.777+0000 7f6954360700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f694c065ce0 0x7f694c10f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:59.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.777+0000 7f6954360700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694c10f810 con 0x7f694c065ce0 2024-04-22T21:01:59.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.777+0000 7f69520fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f694c065ce0 0x7f694c10f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:59.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.777+0000 7f69520fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f694c065ce0 0x7f694c10f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53630/0 (socket says 172.21.15.38:53630) 2024-04-22T21:01:59.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.777+0000 7f69520fc700 1 -- 172.21.15.38:0/3416013377 learned_addr learned my addr 172.21.15.38:0/3416013377 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:01:59.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.777+0000 7f69520fc700 1 -- 172.21.15.38:0/3416013377 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6944009940 con 0x7f694c065ce0 2024-04-22T21:01:59.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.777+0000 7f69520fc700 1 --2- 172.21.15.38:0/3416013377 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f694c065ce0 0x7f694c10f2d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f69440097b0 tx=0x7f6944004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:59.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.778+0000 7f694affd700 1 -- 172.21.15.38:0/3416013377 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6944012410 con 0x7f694c065ce0 2024-04-22T21:01:59.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.778+0000 7f694affd700 1 -- 172.21.15.38:0/3416013377 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6944012a40 con 0x7f694c065ce0 2024-04-22T21:01:59.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.778+0000 7f6954360700 1 -- 172.21.15.38:0/3416013377 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694c10fa10 con 0x7f694c065ce0 2024-04-22T21:01:59.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.778+0000 7f694affd700 1 -- 172.21.15.38:0/3416013377 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f694401b980 con 0x7f694c065ce0 2024-04-22T21:01:59.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.778+0000 7f6954360700 1 -- 172.21.15.38:0/3416013377 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f694c110580 con 0x7f694c065ce0 2024-04-22T21:01:59.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.779+0000 7f694affd700 1 -- 172.21.15.38:0/3416013377 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f6944019040 con 0x7f694c065ce0 2024-04-22T21:01:59.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.779+0000 7f6954360700 1 -- 172.21.15.38:0/3416013377 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f694c105180 con 0x7f694c065ce0 2024-04-22T21:01:59.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.779+0000 7f694affd700 1 --2- 172.21.15.38:0/3416013377 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f69340368d0 0x7f6934038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:01:59.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.779+0000 7f694affd700 1 -- 172.21.15.38:0/3416013377 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6944049c50 con 0x7f694c065ce0 2024-04-22T21:01:59.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.779+0000 7f69518fb700 1 --2- 172.21.15.38:0/3416013377 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f69340368d0 0x7f6934038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:01:59.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.779+0000 7f69518fb700 1 --2- 172.21.15.38:0/3416013377 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f69340368d0 0x7f6934038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6940009990 tx=0x7f6940006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:01:59.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.784+0000 7f694affd700 1 -- 172.21.15.38:0/3416013377 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6944022b10 con 0x7f694c065ce0 2024-04-22T21:01:59.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:01:59.985+0000 7f6954360700 1 -- 172.21.15.38:0/3416013377 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f694c110f90 con 0x7f694c065ce0 2024-04-22T21:02:00.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:02:00.102+0000 7f694affd700 1 -- 172.21.15.38:0/3416013377 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6944017090 con 0x7f694c065ce0 2024-04-22T21:02:00.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:02:00.105+0000 7f6954360700 1 -- 172.21.15.38:0/3416013377 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f69340368d0 msgr2=0x7f6934038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:00.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:02:00.105+0000 7f6954360700 1 --2- 172.21.15.38:0/3416013377 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f69340368d0 0x7f6934038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6940009990 tx=0x7f6940006e40 comp rx=0 tx=0).stop 2024-04-22T21:02:00.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:02:00.105+0000 7f6954360700 1 -- 172.21.15.38:0/3416013377 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f694c065ce0 msgr2=0x7f694c10f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:00.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:02:00.105+0000 7f6954360700 1 --2- 172.21.15.38:0/3416013377 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f694c065ce0 0x7f694c10f2d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f69440097b0 tx=0x7f6944004d10 comp rx=0 tx=0).stop 2024-04-22T21:02:00.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:02:00.105+0000 7f6954360700 1 -- 172.21.15.38:0/3416013377 shutdown_connections 2024-04-22T21:02:00.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:02:00.105+0000 7f6954360700 1 --2- 172.21.15.38:0/3416013377 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f69340368d0 0x7f6934038d80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:00.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:02:00.105+0000 7f6954360700 1 --2- 172.21.15.38:0/3416013377 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f694c065ce0 0x7f694c10f2d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:00.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:02:00.105+0000 7f6954360700 1 -- 172.21.15.38:0/3416013377 >> 172.21.15.38:0/3416013377 conn(0x7f694c0f70d0 msgr2=0x7f694c105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:00.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:02:00.105+0000 7f6954360700 1 -- 172.21.15.38:0/3416013377 shutdown_connections 2024-04-22T21:02:00.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:02:00.105+0000 7f6954360700 1 -- 172.21.15.38:0/3416013377 wait complete. 2024-04-22T21:02:00.102 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2024-04-22T21:02:00.331 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:00 smithi038 ceph-mon[26374]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:00.332 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:00 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3416013377' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T21:02:01.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:01 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3416013377' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T21:02:01.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:01 smithi038 ceph-mon[26374]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T21:02:01.523 INFO:tasks.cephadm:Adding mon.smithi038 on smithi038 2024-04-22T21:02:01.524 INFO:tasks.cephadm:Adding mon.smithi098 on smithi098 2024-04-22T21:02:01.524 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph orch apply mon '2;smithi038:172.21.15.38=smithi038;smithi098:172.21.15.98=smithi098' 2024-04-22T21:02:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:01.833 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:02.187 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:02 smithi038 ceph-mon[26374]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.239+0000 7f6eb7fce700 1 -- 172.21.15.98:0/4115310306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb00669d0 msgr2=0x7f6eb00fd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.239+0000 7f6eb7fce700 1 --2- 172.21.15.98:0/4115310306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb00669d0 0x7f6eb00fd450 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0048c0 tx=0x7f6eac009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.239+0000 7f6eb7fce700 1 -- 172.21.15.98:0/4115310306 shutdown_connections 2024-04-22T21:02:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.239+0000 7f6eb7fce700 1 --2- 172.21.15.98:0/4115310306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb00669d0 0x7f6eb00fd450 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.239+0000 7f6eb7fce700 1 -- 172.21.15.98:0/4115310306 >> 172.21.15.98:0/4115310306 conn(0x7f6eb00f70d0 msgr2=0x7f6eb00f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.239+0000 7f6eb7fce700 1 -- 172.21.15.98:0/4115310306 shutdown_connections 2024-04-22T21:02:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.239+0000 7f6eb7fce700 1 -- 172.21.15.98:0/4115310306 wait complete. 2024-04-22T21:02:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.239+0000 7f6eb7fce700 1 Processor -- start 2024-04-22T21:02:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.239+0000 7f6eb7fce700 1 -- start start 2024-04-22T21:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.240+0000 7f6eb7fce700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb00669d0 0x7f6eb01136e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.240+0000 7f6eb7fce700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb0113c20 con 0x7f6eb00669d0 2024-04-22T21:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.240+0000 7f6eb5d6a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb00669d0 0x7f6eb01136e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.240+0000 7f6eb5d6a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb00669d0 0x7f6eb01136e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:59902/0 (socket says 172.21.15.98:59902) 2024-04-22T21:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.240+0000 7f6eb5d6a700 1 -- 172.21.15.98:0/780805626 learned_addr learned my addr 172.21.15.98:0/780805626 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.240+0000 7f6eb5d6a700 1 -- 172.21.15.98:0/780805626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eac005040 con 0x7f6eb00669d0 2024-04-22T21:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.240+0000 7f6eb5d6a700 1 --2- 172.21.15.98:0/780805626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb00669d0 0x7f6eb01136e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6eac00b900 tx=0x7f6eac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.240+0000 7f6ea67fc700 1 -- 172.21.15.98:0/780805626 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6eac00b5c0 con 0x7f6eb00669d0 2024-04-22T21:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.240+0000 7f6ea67fc700 1 -- 172.21.15.98:0/780805626 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6eac00be70 con 0x7f6eb00669d0 2024-04-22T21:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.240+0000 7f6eb7fce700 1 -- 172.21.15.98:0/780805626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb0113e20 con 0x7f6eb00669d0 2024-04-22T21:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.240+0000 7f6ea67fc700 1 -- 172.21.15.98:0/780805626 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6eac01c590 con 0x7f6eb00669d0 2024-04-22T21:02:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.240+0000 7f6eb7fce700 1 -- 172.21.15.98:0/780805626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb0114990 con 0x7f6eb00669d0 2024-04-22T21:02:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.241+0000 7f6ea67fc700 1 -- 172.21.15.98:0/780805626 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f6eac013420 con 0x7f6eb00669d0 2024-04-22T21:02:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.241+0000 7f6eb7fce700 1 -- 172.21.15.98:0/780805626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb0100d60 con 0x7f6eb00669d0 2024-04-22T21:02:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.241+0000 7f6ea67fc700 1 --2- 172.21.15.98:0/780805626 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f6e9c036880 0x7f6e9c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.241+0000 7f6ea67fc700 1 -- 172.21.15.98:0/780805626 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6eac049410 con 0x7f6eb00669d0 2024-04-22T21:02:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.242+0000 7f6eb5569700 1 --2- 172.21.15.98:0/780805626 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f6e9c036880 0x7f6e9c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.242+0000 7f6eb5569700 1 --2- 172.21.15.98:0/780805626 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f6e9c036880 0x7f6e9c038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009990 tx=0x7f6ea0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:03.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.246+0000 7f6ea67fc700 1 -- 172.21.15.98:0/780805626 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6eac01b350 con 0x7f6eb00669d0 2024-04-22T21:02:03.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.440+0000 7f6eb7fce700 1 -- 172.21.15.98:0/780805626 --> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi038:172.21.15.38=smithi038;smithi098:172.21.15.98=smithi098", "target": ["mon-mgr", ""]}) v1 -- 0x7f6eb0114c80 con 0x7f6e9c036880 2024-04-22T21:02:03.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.445+0000 7f6ea67fc700 1 -- 172.21.15.98:0/780805626 <== mgr.14162 v2:172.21.15.38:6800/989134174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6eb0114c80 con 0x7f6e9c036880 2024-04-22T21:02:03.442 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled mon update... 2024-04-22T21:02:03.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.447+0000 7f6eb7fce700 1 -- 172.21.15.98:0/780805626 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f6e9c036880 msgr2=0x7f6e9c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:03.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.447+0000 7f6eb7fce700 1 --2- 172.21.15.98:0/780805626 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f6e9c036880 0x7f6e9c038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009990 tx=0x7f6ea0006e40 comp rx=0 tx=0).stop 2024-04-22T21:02:03.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.447+0000 7f6eb7fce700 1 -- 172.21.15.98:0/780805626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb00669d0 msgr2=0x7f6eb01136e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:03.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.447+0000 7f6eb7fce700 1 --2- 172.21.15.98:0/780805626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb00669d0 0x7f6eb01136e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6eac00b900 tx=0x7f6eac009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:03.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.447+0000 7f6eb7fce700 1 -- 172.21.15.98:0/780805626 shutdown_connections 2024-04-22T21:02:03.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.447+0000 7f6eb7fce700 1 --2- 172.21.15.98:0/780805626 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f6e9c036880 0x7f6e9c038d30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:03.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.447+0000 7f6eb7fce700 1 --2- 172.21.15.98:0/780805626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb00669d0 0x7f6eb01136e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:03.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.447+0000 7f6eb7fce700 1 -- 172.21.15.98:0/780805626 >> 172.21.15.98:0/780805626 conn(0x7f6eb00f70d0 msgr2=0x7f6eb010a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:03.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.448+0000 7f6eb7fce700 1 -- 172.21.15.98:0/780805626 shutdown_connections 2024-04-22T21:02:03.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:03.448+0000 7f6eb7fce700 1 -- 172.21.15.98:0/780805626 wait complete. 2024-04-22T21:02:04.049 DEBUG:teuthology.orchestra.run.smithi098:mon.smithi098> sudo journalctl -f -n 0 -u ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@mon.smithi098.service 2024-04-22T21:02:04.052 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:02:04.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:02:04.325 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:04.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:04 smithi038 ceph-mon[26374]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:04.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:04 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:04.379 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:05 smithi038 ceph-mon[26374]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi038:172.21.15.38=smithi038;smithi098:172.21.15.98=smithi098", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:02:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:05 smithi038 ceph-mon[26374]: Saving service mon spec with placement smithi038:172.21.15.38=smithi038;smithi098:172.21.15.98=smithi098;count:2 2024-04-22T21:02:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:05 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:05 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:05 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:05.626 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:05 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:05.626 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:05 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:05.626 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:05 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:05.626 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:05 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T21:02:05.626 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:05 smithi038 ceph-mon[26374]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T21:02:05.626 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:05 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:05.626 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:05 smithi038 ceph-mon[26374]: Deploying daemon grafana.smithi038 on smithi038 2024-04-22T21:02:05.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.740+0000 7f57759ab700 1 -- 172.21.15.98:0/1169448089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57700fb960 msgr2=0x7f57700fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:05.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.740+0000 7f57759ab700 1 --2- 172.21.15.98:0/1169448089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57700fb960 0x7f57700fbd70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f57640048c0 tx=0x7f5764009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:05.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.741+0000 7f57759ab700 1 -- 172.21.15.98:0/1169448089 shutdown_connections 2024-04-22T21:02:05.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.741+0000 7f57759ab700 1 --2- 172.21.15.98:0/1169448089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57700fb960 0x7f57700fbd70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:05.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.741+0000 7f57759ab700 1 -- 172.21.15.98:0/1169448089 >> 172.21.15.98:0/1169448089 conn(0x7f57700f70c0 msgr2=0x7f57700f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:05.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.741+0000 7f57759ab700 1 -- 172.21.15.98:0/1169448089 shutdown_connections 2024-04-22T21:02:05.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.741+0000 7f57759ab700 1 -- 172.21.15.98:0/1169448089 wait complete. 2024-04-22T21:02:05.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.741+0000 7f57759ab700 1 Processor -- start 2024-04-22T21:02:05.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.741+0000 7f57759ab700 1 -- start start 2024-04-22T21:02:05.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.742+0000 7f57759ab700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57700fb960 0x7f577010d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:05.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.742+0000 7f57759ab700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577010d5e0 con 0x7f57700fb960 2024-04-22T21:02:05.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.742+0000 7f576effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57700fb960 0x7f577010d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:05.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.742+0000 7f576effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57700fb960 0x7f577010d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:59932/0 (socket says 172.21.15.98:59932) 2024-04-22T21:02:05.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.742+0000 7f576effd700 1 -- 172.21.15.98:0/1600156776 learned_addr learned my addr 172.21.15.98:0/1600156776 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:02:05.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.742+0000 7f576effd700 1 -- 172.21.15.98:0/1600156776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5764005040 con 0x7f57700fb960 2024-04-22T21:02:05.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.742+0000 7f576effd700 1 --2- 172.21.15.98:0/1600156776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57700fb960 0x7f577010d0a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f57640096b0 tx=0x7f5764009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:05.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.742+0000 7f57577fe700 1 -- 172.21.15.98:0/1600156776 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f576400bed0 con 0x7f57700fb960 2024-04-22T21:02:05.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.742+0000 7f57577fe700 1 -- 172.21.15.98:0/1600156776 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f576400b5c0 con 0x7f57700fb960 2024-04-22T21:02:05.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.742+0000 7f57759ab700 1 -- 172.21.15.98:0/1600156776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f577010d7e0 con 0x7f57700fb960 2024-04-22T21:02:05.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.742+0000 7f57577fe700 1 -- 172.21.15.98:0/1600156776 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f576401c5b0 con 0x7f57700fb960 2024-04-22T21:02:05.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.743+0000 7f57759ab700 1 -- 172.21.15.98:0/1600156776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f577010e350 con 0x7f57700fb960 2024-04-22T21:02:05.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.743+0000 7f57577fe700 1 -- 172.21.15.98:0/1600156776 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f5764004d30 con 0x7f57700fb960 2024-04-22T21:02:05.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.743+0000 7f57759ab700 1 -- 172.21.15.98:0/1600156776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5770101e70 con 0x7f57700fb960 2024-04-22T21:02:05.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.744+0000 7f57577fe700 1 --2- 172.21.15.98:0/1600156776 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f5758036880 0x7f5758038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:05.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.744+0000 7f57577fe700 1 -- 172.21.15.98:0/1600156776 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5764049a50 con 0x7f57700fb960 2024-04-22T21:02:05.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.744+0000 7f576e7fc700 1 --2- 172.21.15.98:0/1600156776 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f5758036880 0x7f5758038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:05.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.744+0000 7f576e7fc700 1 --2- 172.21.15.98:0/1600156776 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f5758036880 0x7f5758038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5760009990 tx=0x7f5760006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:05.749+0000 7f57577fe700 1 -- 172.21.15.98:0/1600156776 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f5764011940 con 0x7f57700fb960 2024-04-22T21:02:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:06.013+0000 7f57759ab700 1 -- 172.21.15.98:0/1600156776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f577010e590 con 0x7f57700fb960 2024-04-22T21:02:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:06.014+0000 7f57577fe700 1 -- 172.21.15.98:0/1600156776 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f5764011370 con 0x7f57700fb960 2024-04-22T21:02:06.011 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:02:06.011 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:00:19.700503Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:02:06.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:06.016+0000 7f57759ab700 1 -- 172.21.15.98:0/1600156776 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f5758036880 msgr2=0x7f5758038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:06.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:06.016+0000 7f57759ab700 1 --2- 172.21.15.98:0/1600156776 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f5758036880 0x7f5758038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5760009990 tx=0x7f5760006e40 comp rx=0 tx=0).stop 2024-04-22T21:02:06.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:06.016+0000 7f57759ab700 1 -- 172.21.15.98:0/1600156776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57700fb960 msgr2=0x7f577010d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:06.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:06.016+0000 7f57759ab700 1 --2- 172.21.15.98:0/1600156776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57700fb960 0x7f577010d0a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f57640096b0 tx=0x7f5764009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:06.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:06.016+0000 7f57759ab700 1 -- 172.21.15.98:0/1600156776 shutdown_connections 2024-04-22T21:02:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:06.016+0000 7f57759ab700 1 --2- 172.21.15.98:0/1600156776 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f5758036880 0x7f5758038d30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:06.016+0000 7f57759ab700 1 --2- 172.21.15.98:0/1600156776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57700fb960 0x7f577010d0a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:06.016+0000 7f57759ab700 1 -- 172.21.15.98:0/1600156776 >> 172.21.15.98:0/1600156776 conn(0x7f57700f70c0 msgr2=0x7f5770065d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:06.016+0000 7f57759ab700 1 -- 172.21.15.98:0/1600156776 shutdown_connections 2024-04-22T21:02:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:06.016+0000 7f57759ab700 1 -- 172.21.15.98:0/1600156776 wait complete. 2024-04-22T21:02:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-22T21:02:06.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:06 smithi038 ceph-mon[26374]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:06.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:06 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/1600156776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:02:06.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:06 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:07.536 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:02:07.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:02:07.789 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:07.837 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:08.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:08 smithi038 ceph-mon[26374]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:09.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.238+0000 7f5c0106f700 1 -- 172.21.15.98:0/2206019103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc0fc9f0 msgr2=0x7f5bfc0fce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:09.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.238+0000 7f5c0106f700 1 --2- 172.21.15.98:0/2206019103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc0fc9f0 0x7f5bfc0fce00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5be40048c0 tx=0x7f5be4009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:09.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.238+0000 7f5c0106f700 1 -- 172.21.15.98:0/2206019103 shutdown_connections 2024-04-22T21:02:09.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.238+0000 7f5c0106f700 1 --2- 172.21.15.98:0/2206019103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc0fc9f0 0x7f5bfc0fce00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:09.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.238+0000 7f5c0106f700 1 -- 172.21.15.98:0/2206019103 >> 172.21.15.98:0/2206019103 conn(0x7f5bfc0f8150 msgr2=0x7f5bfc0fa580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:09.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.238+0000 7f5c0106f700 1 -- 172.21.15.98:0/2206019103 shutdown_connections 2024-04-22T21:02:09.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.238+0000 7f5c0106f700 1 -- 172.21.15.98:0/2206019103 wait complete. 2024-04-22T21:02:09.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.239+0000 7f5c0106f700 1 Processor -- start 2024-04-22T21:02:09.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.239+0000 7f5c0106f700 1 -- start start 2024-04-22T21:02:09.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.239+0000 7f5c0106f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc0fc9f0 0x7f5bfc10d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:09.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.239+0000 7f5c0106f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc10d580 con 0x7f5bfc0fc9f0 2024-04-22T21:02:09.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.239+0000 7f5bfad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc0fc9f0 0x7f5bfc10d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:09.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.239+0000 7f5bfad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc0fc9f0 0x7f5bfc10d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:59952/0 (socket says 172.21.15.98:59952) 2024-04-22T21:02:09.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.239+0000 7f5bfad9d700 1 -- 172.21.15.98:0/3343806550 learned_addr learned my addr 172.21.15.98:0/3343806550 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:02:09.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.239+0000 7f5bfad9d700 1 -- 172.21.15.98:0/3343806550 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be4005040 con 0x7f5bfc0fc9f0 2024-04-22T21:02:09.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.240+0000 7f5bfad9d700 1 --2- 172.21.15.98:0/3343806550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc0fc9f0 0x7f5bfc10d040 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5be40096b0 tx=0x7f5be4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:09.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.240+0000 7f5bf37fe700 1 -- 172.21.15.98:0/3343806550 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5be400bed0 con 0x7f5bfc0fc9f0 2024-04-22T21:02:09.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.240+0000 7f5c0106f700 1 -- 172.21.15.98:0/3343806550 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bfc10d780 con 0x7f5bfc0fc9f0 2024-04-22T21:02:09.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.240+0000 7f5bf37fe700 1 -- 172.21.15.98:0/3343806550 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5be400b5c0 con 0x7f5bfc0fc9f0 2024-04-22T21:02:09.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.240+0000 7f5bf37fe700 1 -- 172.21.15.98:0/3343806550 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5be401c5b0 con 0x7f5bfc0fc9f0 2024-04-22T21:02:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.240+0000 7f5c0106f700 1 -- 172.21.15.98:0/3343806550 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bfc10e2f0 con 0x7f5bfc0fc9f0 2024-04-22T21:02:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.241+0000 7f5bf37fe700 1 -- 172.21.15.98:0/3343806550 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f5be4013420 con 0x7f5bfc0fc9f0 2024-04-22T21:02:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.241+0000 7f5c0106f700 1 -- 172.21.15.98:0/3343806550 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bfc102f00 con 0x7f5bfc0fc9f0 2024-04-22T21:02:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.241+0000 7f5bf37fe700 1 --2- 172.21.15.98:0/3343806550 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f5be8036880 0x7f5be8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.241+0000 7f5bf37fe700 1 -- 172.21.15.98:0/3343806550 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5be4049a50 con 0x7f5bfc0fc9f0 2024-04-22T21:02:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.241+0000 7f5bfa59c700 1 --2- 172.21.15.98:0/3343806550 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f5be8036880 0x7f5be8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.242+0000 7f5bfa59c700 1 --2- 172.21.15.98:0/3343806550 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f5be8036880 0x7f5be8038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5bec009990 tx=0x7f5bec006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.246+0000 7f5bf37fe700 1 -- 172.21.15.98:0/3343806550 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f5be4025400 con 0x7f5bfc0fc9f0 2024-04-22T21:02:09.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.505+0000 7f5c0106f700 1 -- 172.21.15.98:0/3343806550 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5bfc10ed00 con 0x7f5bfc0fc9f0 2024-04-22T21:02:09.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.505+0000 7f5bf37fe700 1 -- 172.21.15.98:0/3343806550 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f5be4004d30 con 0x7f5bfc0fc9f0 2024-04-22T21:02:09.502 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:02:09.502 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:00:19.700503Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:02:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.507+0000 7f5c0106f700 1 -- 172.21.15.98:0/3343806550 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f5be8036880 msgr2=0x7f5be8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.507+0000 7f5c0106f700 1 --2- 172.21.15.98:0/3343806550 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f5be8036880 0x7f5be8038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5bec009990 tx=0x7f5bec006e40 comp rx=0 tx=0).stop 2024-04-22T21:02:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.507+0000 7f5c0106f700 1 -- 172.21.15.98:0/3343806550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc0fc9f0 msgr2=0x7f5bfc10d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.507+0000 7f5c0106f700 1 --2- 172.21.15.98:0/3343806550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc0fc9f0 0x7f5bfc10d040 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5be40096b0 tx=0x7f5be4009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.507+0000 7f5c0106f700 1 -- 172.21.15.98:0/3343806550 shutdown_connections 2024-04-22T21:02:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.507+0000 7f5c0106f700 1 --2- 172.21.15.98:0/3343806550 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f5be8036880 0x7f5be8038d30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.507+0000 7f5c0106f700 1 --2- 172.21.15.98:0/3343806550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc0fc9f0 0x7f5bfc10d040 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.507+0000 7f5c0106f700 1 -- 172.21.15.98:0/3343806550 >> 172.21.15.98:0/3343806550 conn(0x7f5bfc0f8150 msgr2=0x7f5bfc103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.507+0000 7f5c0106f700 1 -- 172.21.15.98:0/3343806550 shutdown_connections 2024-04-22T21:02:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:09.507+0000 7f5c0106f700 1 -- 172.21.15.98:0/3343806550 wait complete. 2024-04-22T21:02:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-22T21:02:10.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:10 smithi038 ceph-mon[26374]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:10.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:10 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/3343806550' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:02:11.036 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:02:11.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:02:11.293 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:12.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:12 smithi038 ceph-mon[26374]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.706+0000 7f6f9bc93700 1 -- 172.21.15.98:0/4067978337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f940fca70 msgr2=0x7f6f940fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.706+0000 7f6f9bc93700 1 --2- 172.21.15.98:0/4067978337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f940fca70 0x7f6f940fce80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6f900048c0 tx=0x7f6f90009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.706+0000 7f6f9bc93700 1 -- 172.21.15.98:0/4067978337 shutdown_connections 2024-04-22T21:02:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.706+0000 7f6f9bc93700 1 --2- 172.21.15.98:0/4067978337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f940fca70 0x7f6f940fce80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.706+0000 7f6f9bc93700 1 -- 172.21.15.98:0/4067978337 >> 172.21.15.98:0/4067978337 conn(0x7f6f940f81d0 msgr2=0x7f6f940fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.707+0000 7f6f9bc93700 1 -- 172.21.15.98:0/4067978337 shutdown_connections 2024-04-22T21:02:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.707+0000 7f6f9bc93700 1 -- 172.21.15.98:0/4067978337 wait complete. 2024-04-22T21:02:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.707+0000 7f6f9bc93700 1 Processor -- start 2024-04-22T21:02:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.707+0000 7f6f9bc93700 1 -- start start 2024-04-22T21:02:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.707+0000 7f6f9bc93700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f940fca70 0x7f6f94071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.707+0000 7f6f9bc93700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f940722a0 con 0x7f6f940fca70 2024-04-22T21:02:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.707+0000 7f6f99a2f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f940fca70 0x7f6f94071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.708+0000 7f6f99a2f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f940fca70 0x7f6f94071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:34998/0 (socket says 172.21.15.98:34998) 2024-04-22T21:02:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.708+0000 7f6f99a2f700 1 -- 172.21.15.98:0/2123601850 learned_addr learned my addr 172.21.15.98:0/2123601850 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:02:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.708+0000 7f6f99a2f700 1 -- 172.21.15.98:0/2123601850 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f90005040 con 0x7f6f940fca70 2024-04-22T21:02:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.708+0000 7f6f99a2f700 1 --2- 172.21.15.98:0/2123601850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f940fca70 0x7f6f94071d60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6f900096b0 tx=0x7f6f90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.708+0000 7f6f8a7fc700 1 -- 172.21.15.98:0/2123601850 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f9000bed0 con 0x7f6f940fca70 2024-04-22T21:02:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.708+0000 7f6f9bc93700 1 -- 172.21.15.98:0/2123601850 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f94070370 con 0x7f6f940fca70 2024-04-22T21:02:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.708+0000 7f6f8a7fc700 1 -- 172.21.15.98:0/2123601850 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f9000b5c0 con 0x7f6f940fca70 2024-04-22T21:02:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.708+0000 7f6f8a7fc700 1 -- 172.21.15.98:0/2123601850 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f9001c5b0 con 0x7f6f940fca70 2024-04-22T21:02:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.708+0000 7f6f9bc93700 1 -- 172.21.15.98:0/2123601850 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f94070860 con 0x7f6f940fca70 2024-04-22T21:02:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.709+0000 7f6f8a7fc700 1 -- 172.21.15.98:0/2123601850 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f6f90013420 con 0x7f6f940fca70 2024-04-22T21:02:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.709+0000 7f6f9bc93700 1 -- 172.21.15.98:0/2123601850 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9405f5f0 con 0x7f6f940fca70 2024-04-22T21:02:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.709+0000 7f6f8a7fc700 1 --2- 172.21.15.98:0/2123601850 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f6f80036880 0x7f6f80038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.709+0000 7f6f8a7fc700 1 -- 172.21.15.98:0/2123601850 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6f90049ac0 con 0x7f6f940fca70 2024-04-22T21:02:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.709+0000 7f6f9922e700 1 --2- 172.21.15.98:0/2123601850 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f6f80036880 0x7f6f80038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:12.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.710+0000 7f6f9922e700 1 --2- 172.21.15.98:0/2123601850 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f6f80036880 0x7f6f80038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009990 tx=0x7f6f84006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:12.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.718+0000 7f6f8a7fc700 1 -- 172.21.15.98:0/2123601850 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6f90013a80 con 0x7f6f940fca70 2024-04-22T21:02:12.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.976+0000 7f6f9bc93700 1 -- 172.21.15.98:0/2123601850 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f94106450 con 0x7f6f940fca70 2024-04-22T21:02:12.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.977+0000 7f6f8a7fc700 1 -- 172.21.15.98:0/2123601850 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f6f90011370 con 0x7f6f940fca70 2024-04-22T21:02:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:02:12.973 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:00:19.700503Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:02:12.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.979+0000 7f6f9bc93700 1 -- 172.21.15.98:0/2123601850 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f6f80036880 msgr2=0x7f6f80038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:12.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.979+0000 7f6f9bc93700 1 --2- 172.21.15.98:0/2123601850 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f6f80036880 0x7f6f80038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009990 tx=0x7f6f84006e40 comp rx=0 tx=0).stop 2024-04-22T21:02:12.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.979+0000 7f6f9bc93700 1 -- 172.21.15.98:0/2123601850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f940fca70 msgr2=0x7f6f94071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:12.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.979+0000 7f6f9bc93700 1 --2- 172.21.15.98:0/2123601850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f940fca70 0x7f6f94071d60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6f900096b0 tx=0x7f6f90009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:12.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.979+0000 7f6f9bc93700 1 -- 172.21.15.98:0/2123601850 shutdown_connections 2024-04-22T21:02:12.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.979+0000 7f6f9bc93700 1 --2- 172.21.15.98:0/2123601850 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f6f80036880 0x7f6f80038d30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:12.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.979+0000 7f6f9bc93700 1 --2- 172.21.15.98:0/2123601850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f940fca70 0x7f6f94071d60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:12.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.979+0000 7f6f9bc93700 1 -- 172.21.15.98:0/2123601850 >> 172.21.15.98:0/2123601850 conn(0x7f6f940f81d0 msgr2=0x7f6f94103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:12.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.979+0000 7f6f9bc93700 1 -- 172.21.15.98:0/2123601850 shutdown_connections 2024-04-22T21:02:12.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:12.979+0000 7f6f9bc93700 1 -- 172.21.15.98:0/2123601850 wait complete. 2024-04-22T21:02:12.976 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-22T21:02:13.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:13 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/2123601850' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:02:14.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:14 smithi038 ceph-mon[26374]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:14.511 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:02:14.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:02:14.761 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:14.807 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:16.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.256+0000 7ff765880700 1 -- 172.21.15.98:0/2977173557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7600fd640 msgr2=0x7ff7600ffa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:16.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.256+0000 7ff765880700 1 --2- 172.21.15.98:0/2977173557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7600fd640 0x7ff7600ffa20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff7500048c0 tx=0x7ff750009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:16.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.256+0000 7ff765880700 1 -- 172.21.15.98:0/2977173557 shutdown_connections 2024-04-22T21:02:16.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.256+0000 7ff765880700 1 --2- 172.21.15.98:0/2977173557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7600fd640 0x7ff7600ffa20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:16.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.256+0000 7ff765880700 1 -- 172.21.15.98:0/2977173557 >> 172.21.15.98:0/2977173557 conn(0x7ff7600f70c0 msgr2=0x7ff7600f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:16.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.256+0000 7ff765880700 1 -- 172.21.15.98:0/2977173557 shutdown_connections 2024-04-22T21:02:16.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.256+0000 7ff765880700 1 -- 172.21.15.98:0/2977173557 wait complete. 2024-04-22T21:02:16.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.257+0000 7ff765880700 1 Processor -- start 2024-04-22T21:02:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.257+0000 7ff765880700 1 -- start start 2024-04-22T21:02:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.257+0000 7ff765880700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7600fd640 0x7ff76010d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.257+0000 7ff765880700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76010d6a0 con 0x7ff7600fd640 2024-04-22T21:02:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.258+0000 7ff75effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7600fd640 0x7ff76010d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.258+0000 7ff75effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7600fd640 0x7ff76010d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:35018/0 (socket says 172.21.15.98:35018) 2024-04-22T21:02:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.258+0000 7ff75effd700 1 -- 172.21.15.98:0/454117421 learned_addr learned my addr 172.21.15.98:0/454117421 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:02:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.258+0000 7ff75effd700 1 -- 172.21.15.98:0/454117421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff750005040 con 0x7ff7600fd640 2024-04-22T21:02:16.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.258+0000 7ff75effd700 1 --2- 172.21.15.98:0/454117421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7600fd640 0x7ff76010d160 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff75000b900 tx=0x7ff750009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:16.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.258+0000 7ff7477fe700 1 -- 172.21.15.98:0/454117421 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff75000b5c0 con 0x7ff7600fd640 2024-04-22T21:02:16.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.258+0000 7ff765880700 1 -- 172.21.15.98:0/454117421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff76010d8a0 con 0x7ff7600fd640 2024-04-22T21:02:16.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.258+0000 7ff7477fe700 1 -- 172.21.15.98:0/454117421 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff75000be70 con 0x7ff7600fd640 2024-04-22T21:02:16.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.258+0000 7ff7477fe700 1 -- 172.21.15.98:0/454117421 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff75001c590 con 0x7ff7600fd640 2024-04-22T21:02:16.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.258+0000 7ff765880700 1 -- 172.21.15.98:0/454117421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff76010e410 con 0x7ff7600fd640 2024-04-22T21:02:16.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.259+0000 7ff7477fe700 1 -- 172.21.15.98:0/454117421 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7ff750013420 con 0x7ff7600fd640 2024-04-22T21:02:16.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.259+0000 7ff765880700 1 -- 172.21.15.98:0/454117421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff760102f50 con 0x7ff7600fd640 2024-04-22T21:02:16.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.259+0000 7ff7477fe700 1 --2- 172.21.15.98:0/454117421 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7ff748036880 0x7ff748038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:16.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.259+0000 7ff7477fe700 1 -- 172.21.15.98:0/454117421 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff7500493c0 con 0x7ff7600fd640 2024-04-22T21:02:16.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.259+0000 7ff75e7fc700 1 --2- 172.21.15.98:0/454117421 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7ff748036880 0x7ff748038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:16.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.260+0000 7ff75e7fc700 1 --2- 172.21.15.98:0/454117421 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7ff748036880 0x7ff748038d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff754009990 tx=0x7ff754006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:16.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.264+0000 7ff7477fe700 1 -- 172.21.15.98:0/454117421 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff75001b350 con 0x7ff7600fd640 2024-04-22T21:02:16.501 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:16 smithi038 ceph-mon[26374]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:16.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.523+0000 7ff765880700 1 -- 172.21.15.98:0/454117421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff76010ee20 con 0x7ff7600fd640 2024-04-22T21:02:16.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.523+0000 7ff7477fe700 1 -- 172.21.15.98:0/454117421 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ff750021020 con 0x7ff7600fd640 2024-04-22T21:02:16.520 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:02:16.520 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:00:19.700503Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:02:16.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.525+0000 7ff765880700 1 -- 172.21.15.98:0/454117421 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7ff748036880 msgr2=0x7ff748038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:16.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.525+0000 7ff765880700 1 --2- 172.21.15.98:0/454117421 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7ff748036880 0x7ff748038d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff754009990 tx=0x7ff754006e40 comp rx=0 tx=0).stop 2024-04-22T21:02:16.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.525+0000 7ff765880700 1 -- 172.21.15.98:0/454117421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7600fd640 msgr2=0x7ff76010d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:16.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.525+0000 7ff765880700 1 --2- 172.21.15.98:0/454117421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7600fd640 0x7ff76010d160 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff75000b900 tx=0x7ff750009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:16.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.525+0000 7ff765880700 1 -- 172.21.15.98:0/454117421 shutdown_connections 2024-04-22T21:02:16.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.525+0000 7ff765880700 1 --2- 172.21.15.98:0/454117421 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7ff748036880 0x7ff748038d30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:16.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.525+0000 7ff765880700 1 --2- 172.21.15.98:0/454117421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7600fd640 0x7ff76010d160 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:16.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.525+0000 7ff765880700 1 -- 172.21.15.98:0/454117421 >> 172.21.15.98:0/454117421 conn(0x7ff7600f70c0 msgr2=0x7ff760103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:16.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.525+0000 7ff765880700 1 -- 172.21.15.98:0/454117421 shutdown_connections 2024-04-22T21:02:16.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:16.525+0000 7ff765880700 1 -- 172.21.15.98:0/454117421 wait complete. 2024-04-22T21:02:16.523 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-22T21:02:17.511 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:17 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/454117421' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:02:18.107 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:02:18.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:02:18.362 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:18.370 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:18 smithi038 ceph-mon[26374]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:19.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.713+0000 7fb541549700 1 -- 172.21.15.98:0/2791758660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb53c0fb960 msgr2=0x7fb53c0fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:19.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.713+0000 7fb541549700 1 --2- 172.21.15.98:0/2791758660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb53c0fb960 0x7fb53c0fbd70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb5300048c0 tx=0x7fb530009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:19.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.713+0000 7fb541549700 1 -- 172.21.15.98:0/2791758660 shutdown_connections 2024-04-22T21:02:19.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.713+0000 7fb541549700 1 --2- 172.21.15.98:0/2791758660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb53c0fb960 0x7fb53c0fbd70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:19.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.713+0000 7fb541549700 1 -- 172.21.15.98:0/2791758660 >> 172.21.15.98:0/2791758660 conn(0x7fb53c0f70c0 msgr2=0x7fb53c0f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:19.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.713+0000 7fb541549700 1 -- 172.21.15.98:0/2791758660 shutdown_connections 2024-04-22T21:02:19.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.713+0000 7fb541549700 1 -- 172.21.15.98:0/2791758660 wait complete. 2024-04-22T21:02:19.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.714+0000 7fb541549700 1 Processor -- start 2024-04-22T21:02:19.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.714+0000 7fb541549700 1 -- start start 2024-04-22T21:02:19.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.714+0000 7fb541549700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb53c0fb960 0x7fb53c0667f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:19.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.714+0000 7fb541549700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c10fdd0 con 0x7fb53c0fb960 2024-04-22T21:02:19.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.714+0000 7fb53affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb53c0fb960 0x7fb53c0667f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:19.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.714+0000 7fb53affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb53c0fb960 0x7fb53c0667f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:34532/0 (socket says 172.21.15.98:34532) 2024-04-22T21:02:19.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.714+0000 7fb53affd700 1 -- 172.21.15.98:0/3153398667 learned_addr learned my addr 172.21.15.98:0/3153398667 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:02:19.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.715+0000 7fb53affd700 1 -- 172.21.15.98:0/3153398667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb530005040 con 0x7fb53c0fb960 2024-04-22T21:02:19.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.715+0000 7fb53affd700 1 --2- 172.21.15.98:0/3153398667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb53c0fb960 0x7fb53c0667f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb5300096b0 tx=0x7fb530009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:19.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.715+0000 7fb5237fe700 1 -- 172.21.15.98:0/3153398667 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb53000bed0 con 0x7fb53c0fb960 2024-04-22T21:02:19.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.715+0000 7fb541549700 1 -- 172.21.15.98:0/3153398667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb53c10ff10 con 0x7fb53c0fb960 2024-04-22T21:02:19.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.715+0000 7fb5237fe700 1 -- 172.21.15.98:0/3153398667 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb53000b5c0 con 0x7fb53c0fb960 2024-04-22T21:02:19.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.715+0000 7fb5237fe700 1 -- 172.21.15.98:0/3153398667 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb53001c5b0 con 0x7fb53c0fb960 2024-04-22T21:02:19.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.715+0000 7fb541549700 1 -- 172.21.15.98:0/3153398667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb53c110400 con 0x7fb53c0fb960 2024-04-22T21:02:19.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.716+0000 7fb5237fe700 1 -- 172.21.15.98:0/3153398667 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fb53001c820 con 0x7fb53c0fb960 2024-04-22T21:02:19.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.716+0000 7fb541549700 1 -- 172.21.15.98:0/3153398667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb53c105920 con 0x7fb53c0fb960 2024-04-22T21:02:19.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.716+0000 7fb5237fe700 1 --2- 172.21.15.98:0/3153398667 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fb524036880 0x7fb524038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:19.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.716+0000 7fb5237fe700 1 -- 172.21.15.98:0/3153398667 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb530049a90 con 0x7fb53c0fb960 2024-04-22T21:02:19.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.717+0000 7fb53a7fc700 1 --2- 172.21.15.98:0/3153398667 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fb524036880 0x7fb524038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:19.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.717+0000 7fb53a7fc700 1 --2- 172.21.15.98:0/3153398667 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fb524036880 0x7fb524038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb52c009990 tx=0x7fb52c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:19.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.724+0000 7fb5237fe700 1 -- 172.21.15.98:0/3153398667 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb530023300 con 0x7fb53c0fb960 2024-04-22T21:02:19.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.984+0000 7fb541549700 1 -- 172.21.15.98:0/3153398667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb53c05f5f0 con 0x7fb53c0fb960 2024-04-22T21:02:19.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.984+0000 7fb5237fe700 1 -- 172.21.15.98:0/3153398667 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb530011b60 con 0x7fb53c0fb960 2024-04-22T21:02:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:02:19.981 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:00:19.700503Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:02:19.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.987+0000 7fb541549700 1 -- 172.21.15.98:0/3153398667 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fb524036880 msgr2=0x7fb524038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:19.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.987+0000 7fb541549700 1 --2- 172.21.15.98:0/3153398667 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fb524036880 0x7fb524038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb52c009990 tx=0x7fb52c006e40 comp rx=0 tx=0).stop 2024-04-22T21:02:19.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.987+0000 7fb541549700 1 -- 172.21.15.98:0/3153398667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb53c0fb960 msgr2=0x7fb53c0667f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:19.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.987+0000 7fb541549700 1 --2- 172.21.15.98:0/3153398667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb53c0fb960 0x7fb53c0667f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb5300096b0 tx=0x7fb530009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:19.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.987+0000 7fb541549700 1 -- 172.21.15.98:0/3153398667 shutdown_connections 2024-04-22T21:02:19.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.987+0000 7fb541549700 1 --2- 172.21.15.98:0/3153398667 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fb524036880 0x7fb524038d30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:19.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.987+0000 7fb541549700 1 --2- 172.21.15.98:0/3153398667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb53c0fb960 0x7fb53c0667f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:19.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.987+0000 7fb541549700 1 -- 172.21.15.98:0/3153398667 >> 172.21.15.98:0/3153398667 conn(0x7fb53c0f70c0 msgr2=0x7fb53c102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:19.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.987+0000 7fb541549700 1 -- 172.21.15.98:0/3153398667 shutdown_connections 2024-04-22T21:02:19.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:19.987+0000 7fb541549700 1 -- 172.21.15.98:0/3153398667 wait complete. 2024-04-22T21:02:19.987 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-22T21:02:20.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:20 smithi038 ceph-mon[26374]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:20.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:20 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:20.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:20 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:20.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:20 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:20.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:20 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:20.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:20 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:20.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:20 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:20.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:20 smithi038 ceph-mon[26374]: Deploying daemon prometheus.smithi038 on smithi038 2024-04-22T21:02:20.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:20 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/3153398667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:02:21.572 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:02:21.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:02:21.821 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:21.868 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:22.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:22 smithi038 ceph-mon[26374]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:22.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:22 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:23.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.338+0000 7effe5160700 1 -- 172.21.15.98:0/184459124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effe00fca60 msgr2=0x7effe00fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:23.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.338+0000 7effe5160700 1 --2- 172.21.15.98:0/184459124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effe00fca60 0x7effe00fce70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7effd40048c0 tx=0x7effd4009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:23.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.338+0000 7effe5160700 1 -- 172.21.15.98:0/184459124 shutdown_connections 2024-04-22T21:02:23.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.338+0000 7effe5160700 1 --2- 172.21.15.98:0/184459124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effe00fca60 0x7effe00fce70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:23.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.338+0000 7effe5160700 1 -- 172.21.15.98:0/184459124 >> 172.21.15.98:0/184459124 conn(0x7effe00f81c0 msgr2=0x7effe00fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:23.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.338+0000 7effe5160700 1 -- 172.21.15.98:0/184459124 shutdown_connections 2024-04-22T21:02:23.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.338+0000 7effe5160700 1 -- 172.21.15.98:0/184459124 wait complete. 2024-04-22T21:02:23.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.339+0000 7effe5160700 1 Processor -- start 2024-04-22T21:02:23.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.339+0000 7effe5160700 1 -- start start 2024-04-22T21:02:23.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.339+0000 7effe5160700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effe00fca60 0x7effe010d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:23.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.339+0000 7effe5160700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe010d5f0 con 0x7effe00fca60 2024-04-22T21:02:23.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.339+0000 7effded9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effe00fca60 0x7effe010d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:23.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.339+0000 7effded9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effe00fca60 0x7effe010d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:34540/0 (socket says 172.21.15.98:34540) 2024-04-22T21:02:23.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.339+0000 7effded9d700 1 -- 172.21.15.98:0/2814541345 learned_addr learned my addr 172.21.15.98:0/2814541345 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:02:23.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.339+0000 7effded9d700 1 -- 172.21.15.98:0/2814541345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd4005040 con 0x7effe00fca60 2024-04-22T21:02:23.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.339+0000 7effded9d700 1 --2- 172.21.15.98:0/2814541345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effe00fca60 0x7effe010d0b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7effd40096b0 tx=0x7effd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:23.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.340+0000 7effc77fe700 1 -- 172.21.15.98:0/2814541345 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effd400bed0 con 0x7effe00fca60 2024-04-22T21:02:23.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.340+0000 7effc77fe700 1 -- 172.21.15.98:0/2814541345 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effd400b5c0 con 0x7effe00fca60 2024-04-22T21:02:23.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.340+0000 7effe5160700 1 -- 172.21.15.98:0/2814541345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe010d7f0 con 0x7effe00fca60 2024-04-22T21:02:23.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.340+0000 7effc77fe700 1 -- 172.21.15.98:0/2814541345 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effd401c5b0 con 0x7effe00fca60 2024-04-22T21:02:23.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.340+0000 7effe5160700 1 -- 172.21.15.98:0/2814541345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe010e360 con 0x7effe00fca60 2024-04-22T21:02:23.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.340+0000 7effc77fe700 1 -- 172.21.15.98:0/2814541345 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7effd401c710 con 0x7effe00fca60 2024-04-22T21:02:23.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.340+0000 7effe5160700 1 -- 172.21.15.98:0/2814541345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe0102f70 con 0x7effe00fca60 2024-04-22T21:02:23.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.341+0000 7effc77fe700 1 --2- 172.21.15.98:0/2814541345 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7effc80368d0 0x7effc8038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:23.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.341+0000 7effc77fe700 1 -- 172.21.15.98:0/2814541345 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7effd4049b70 con 0x7effe00fca60 2024-04-22T21:02:23.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.341+0000 7effde59c700 1 --2- 172.21.15.98:0/2814541345 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7effc80368d0 0x7effc8038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:23.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.341+0000 7effde59c700 1 --2- 172.21.15.98:0/2814541345 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7effc80368d0 0x7effc8038d80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7effd0009990 tx=0x7effd0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:23.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.345+0000 7effc77fe700 1 -- 172.21.15.98:0/2814541345 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7effd4025400 con 0x7effe00fca60 2024-04-22T21:02:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.602+0000 7effe5160700 1 -- 172.21.15.98:0/2814541345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7effe010ed70 con 0x7effe00fca60 2024-04-22T21:02:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.603+0000 7effc77fe700 1 -- 172.21.15.98:0/2814541345 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7effd4011370 con 0x7effe00fca60 2024-04-22T21:02:23.599 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:02:23.599 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:00:19.700503Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:02:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.604+0000 7effe5160700 1 -- 172.21.15.98:0/2814541345 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7effc80368d0 msgr2=0x7effc8038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.604+0000 7effe5160700 1 --2- 172.21.15.98:0/2814541345 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7effc80368d0 0x7effc8038d80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7effd0009990 tx=0x7effd0006e40 comp rx=0 tx=0).stop 2024-04-22T21:02:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.604+0000 7effe5160700 1 -- 172.21.15.98:0/2814541345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effe00fca60 msgr2=0x7effe010d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.604+0000 7effe5160700 1 --2- 172.21.15.98:0/2814541345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effe00fca60 0x7effe010d0b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7effd40096b0 tx=0x7effd4009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.604+0000 7effe5160700 1 -- 172.21.15.98:0/2814541345 shutdown_connections 2024-04-22T21:02:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.605+0000 7effe5160700 1 --2- 172.21.15.98:0/2814541345 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7effc80368d0 0x7effc8038d80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.605+0000 7effe5160700 1 --2- 172.21.15.98:0/2814541345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effe00fca60 0x7effe010d0b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.605+0000 7effe5160700 1 -- 172.21.15.98:0/2814541345 >> 172.21.15.98:0/2814541345 conn(0x7effe00f81c0 msgr2=0x7effe0103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.605+0000 7effe5160700 1 -- 172.21.15.98:0/2814541345 shutdown_connections 2024-04-22T21:02:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:23.605+0000 7effe5160700 1 -- 172.21.15.98:0/2814541345 wait complete. 2024-04-22T21:02:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-22T21:02:24.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:24 smithi038 ceph-mon[26374]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:24.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:24 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/2814541345' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:02:25.179 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:02:25.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:02:25.426 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:26.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:26 smithi038 ceph-mon[26374]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:26.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.949+0000 7fa9aad3a700 1 -- 172.21.15.98:0/2525876054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a40fca70 msgr2=0x7fa9a40fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:26.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.949+0000 7fa9aad3a700 1 --2- 172.21.15.98:0/2525876054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a40fca70 0x7fa9a40fce80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa99c009c60 tx=0x7fa99c004d10 comp rx=0 tx=0).stop 2024-04-22T21:02:26.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.949+0000 7fa9aad3a700 1 -- 172.21.15.98:0/2525876054 shutdown_connections 2024-04-22T21:02:26.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.949+0000 7fa9aad3a700 1 --2- 172.21.15.98:0/2525876054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a40fca70 0x7fa9a40fce80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:26.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.949+0000 7fa9aad3a700 1 -- 172.21.15.98:0/2525876054 >> 172.21.15.98:0/2525876054 conn(0x7fa9a40f81d0 msgr2=0x7fa9a40fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:26.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.949+0000 7fa9aad3a700 1 -- 172.21.15.98:0/2525876054 shutdown_connections 2024-04-22T21:02:26.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.949+0000 7fa9aad3a700 1 -- 172.21.15.98:0/2525876054 wait complete. 2024-04-22T21:02:26.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.950+0000 7fa9aad3a700 1 Processor -- start 2024-04-22T21:02:26.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.950+0000 7fa9aad3a700 1 -- start start 2024-04-22T21:02:26.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.950+0000 7fa9aad3a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a40fca70 0x7fa9a4071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:26.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.950+0000 7fa9aad3a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a40722a0 con 0x7fa9a40fca70 2024-04-22T21:02:26.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.950+0000 7fa9a8ad6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a40fca70 0x7fa9a4071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:26.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.950+0000 7fa9a8ad6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a40fca70 0x7fa9a4071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:34558/0 (socket says 172.21.15.98:34558) 2024-04-22T21:02:26.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.950+0000 7fa9a8ad6700 1 -- 172.21.15.98:0/158392414 learned_addr learned my addr 172.21.15.98:0/158392414 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:02:26.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.951+0000 7fa9a8ad6700 1 -- 172.21.15.98:0/158392414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c009940 con 0x7fa9a40fca70 2024-04-22T21:02:26.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.951+0000 7fa9a8ad6700 1 --2- 172.21.15.98:0/158392414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a40fca70 0x7fa9a4071d60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa99c009500 tx=0x7fa99c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:26.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.951+0000 7fa9a17fa700 1 -- 172.21.15.98:0/158392414 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa99c007c80 con 0x7fa9a40fca70 2024-04-22T21:02:26.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.951+0000 7fa9a17fa700 1 -- 172.21.15.98:0/158392414 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa99c01d910 con 0x7fa9a40fca70 2024-04-22T21:02:26.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.951+0000 7fa9aad3a700 1 -- 172.21.15.98:0/158392414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a4070370 con 0x7fa9a40fca70 2024-04-22T21:02:26.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.951+0000 7fa9a17fa700 1 -- 172.21.15.98:0/158392414 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa99c0129d0 con 0x7fa9a40fca70 2024-04-22T21:02:26.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.951+0000 7fa9aad3a700 1 -- 172.21.15.98:0/158392414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a4070860 con 0x7fa9a40fca70 2024-04-22T21:02:26.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.952+0000 7fa9aad3a700 1 -- 172.21.15.98:0/158392414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a405f5f0 con 0x7fa9a40fca70 2024-04-22T21:02:26.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.952+0000 7fa9a17fa700 1 -- 172.21.15.98:0/158392414 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fa99c019040 con 0x7fa9a40fca70 2024-04-22T21:02:26.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.952+0000 7fa9a17fa700 1 --2- 172.21.15.98:0/158392414 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fa98c036880 0x7fa98c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:26.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.952+0000 7fa9a17fa700 1 -- 172.21.15.98:0/158392414 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa99c04a390 con 0x7fa9a40fca70 2024-04-22T21:02:26.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.952+0000 7fa9a3fff700 1 --2- 172.21.15.98:0/158392414 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fa98c036880 0x7fa98c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:26.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.953+0000 7fa9a3fff700 1 --2- 172.21.15.98:0/158392414 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fa98c036880 0x7fa98c038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa998009990 tx=0x7fa998006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:26.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:26.957+0000 7fa9a17fa700 1 -- 172.21.15.98:0/158392414 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa99c02eb40 con 0x7fa9a40fca70 2024-04-22T21:02:27.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:27.214+0000 7fa9aad3a700 1 -- 172.21.15.98:0/158392414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9a404bed0 con 0x7fa9a40fca70 2024-04-22T21:02:27.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:27.215+0000 7fa9a17fa700 1 -- 172.21.15.98:0/158392414 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fa99c017020 con 0x7fa9a40fca70 2024-04-22T21:02:27.211 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:02:27.211 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:00:19.700503Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:02:27.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:27.216+0000 7fa9aad3a700 1 -- 172.21.15.98:0/158392414 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fa98c036880 msgr2=0x7fa98c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:27.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:27.216+0000 7fa9aad3a700 1 --2- 172.21.15.98:0/158392414 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fa98c036880 0x7fa98c038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa998009990 tx=0x7fa998006e40 comp rx=0 tx=0).stop 2024-04-22T21:02:27.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:27.216+0000 7fa9aad3a700 1 -- 172.21.15.98:0/158392414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a40fca70 msgr2=0x7fa9a4071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:27.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:27.216+0000 7fa9aad3a700 1 --2- 172.21.15.98:0/158392414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a40fca70 0x7fa9a4071d60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa99c009500 tx=0x7fa99c004d10 comp rx=0 tx=0).stop 2024-04-22T21:02:27.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:27.217+0000 7fa9aad3a700 1 -- 172.21.15.98:0/158392414 shutdown_connections 2024-04-22T21:02:27.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:27.217+0000 7fa9aad3a700 1 --2- 172.21.15.98:0/158392414 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7fa98c036880 0x7fa98c038d30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:27.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:27.217+0000 7fa9aad3a700 1 --2- 172.21.15.98:0/158392414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a40fca70 0x7fa9a4071d60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:27.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:27.217+0000 7fa9aad3a700 1 -- 172.21.15.98:0/158392414 >> 172.21.15.98:0/158392414 conn(0x7fa9a40f81d0 msgr2=0x7fa9a4103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:27.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:27.217+0000 7fa9aad3a700 1 -- 172.21.15.98:0/158392414 shutdown_connections 2024-04-22T21:02:27.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:27.217+0000 7fa9aad3a700 1 -- 172.21.15.98:0/158392414 wait complete. 2024-04-22T21:02:27.214 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-22T21:02:28.305 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:28 smithi038 ceph-mon[26374]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:28.305 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:28 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/158392414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:02:28.739 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:02:28.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:02:28.987 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:29.034 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:30.190 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:30 smithi038 ceph-mon[26374]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:30.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.555+0000 7f57cdec2700 1 -- 172.21.15.98:0/835924461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57c80fb970 msgr2=0x7f57c80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:30.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.555+0000 7f57cdec2700 1 --2- 172.21.15.98:0/835924461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57c80fb970 0x7f57c80fbd80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f57b80048c0 tx=0x7f57b8009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:30.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.555+0000 7f57cdec2700 1 -- 172.21.15.98:0/835924461 shutdown_connections 2024-04-22T21:02:30.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.555+0000 7f57cdec2700 1 --2- 172.21.15.98:0/835924461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57c80fb970 0x7f57c80fbd80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:30.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.555+0000 7f57cdec2700 1 -- 172.21.15.98:0/835924461 >> 172.21.15.98:0/835924461 conn(0x7f57c80f70d0 msgr2=0x7f57c80f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:30.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.555+0000 7f57cdec2700 1 -- 172.21.15.98:0/835924461 shutdown_connections 2024-04-22T21:02:30.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.555+0000 7f57cdec2700 1 -- 172.21.15.98:0/835924461 wait complete. 2024-04-22T21:02:30.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.556+0000 7f57cdec2700 1 Processor -- start 2024-04-22T21:02:30.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.556+0000 7f57cdec2700 1 -- start start 2024-04-22T21:02:30.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.556+0000 7f57cdec2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57c80fb970 0x7f57c8065be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:30.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.556+0000 7f57cdec2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c810fde0 con 0x7f57c80fb970 2024-04-22T21:02:30.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.556+0000 7f57c77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57c80fb970 0x7f57c8065be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:30.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.556+0000 7f57c77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57c80fb970 0x7f57c8065be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:57440/0 (socket says 172.21.15.98:57440) 2024-04-22T21:02:30.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.556+0000 7f57c77fe700 1 -- 172.21.15.98:0/2375417514 learned_addr learned my addr 172.21.15.98:0/2375417514 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:02:30.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.557+0000 7f57c77fe700 1 -- 172.21.15.98:0/2375417514 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b8005040 con 0x7f57c80fb970 2024-04-22T21:02:30.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.557+0000 7f57c77fe700 1 --2- 172.21.15.98:0/2375417514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57c80fb970 0x7f57c8065be0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f57b80096b0 tx=0x7f57b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:30.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.557+0000 7f57c4ff9700 1 -- 172.21.15.98:0/2375417514 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57b800bed0 con 0x7f57c80fb970 2024-04-22T21:02:30.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.557+0000 7f57c4ff9700 1 -- 172.21.15.98:0/2375417514 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57b800b5c0 con 0x7f57c80fb970 2024-04-22T21:02:30.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.557+0000 7f57cdec2700 1 -- 172.21.15.98:0/2375417514 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c810ff20 con 0x7f57c80fb970 2024-04-22T21:02:30.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.557+0000 7f57c4ff9700 1 -- 172.21.15.98:0/2375417514 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57b801c5b0 con 0x7f57c80fb970 2024-04-22T21:02:30.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.557+0000 7f57cdec2700 1 -- 172.21.15.98:0/2375417514 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c8110410 con 0x7f57c80fb970 2024-04-22T21:02:30.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.558+0000 7f57c4ff9700 1 -- 172.21.15.98:0/2375417514 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f57b801c820 con 0x7f57c80fb970 2024-04-22T21:02:30.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.558+0000 7f57cdec2700 1 -- 172.21.15.98:0/2375417514 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c8105930 con 0x7f57c80fb970 2024-04-22T21:02:30.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.558+0000 7f57c4ff9700 1 --2- 172.21.15.98:0/2375417514 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f57b4036880 0x7f57b4038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:30.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.558+0000 7f57c4ff9700 1 -- 172.21.15.98:0/2375417514 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f57b8049bc0 con 0x7f57c80fb970 2024-04-22T21:02:30.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.558+0000 7f57beffd700 1 --2- 172.21.15.98:0/2375417514 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f57b4036880 0x7f57b4038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:30.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.559+0000 7f57beffd700 1 --2- 172.21.15.98:0/2375417514 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f57b4036880 0x7f57b4038d30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009990 tx=0x7f57b0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.563+0000 7f57c4ff9700 1 -- 172.21.15.98:0/2375417514 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f57b8023300 con 0x7f57c80fb970 2024-04-22T21:02:30.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.826+0000 7f57cdec2700 1 -- 172.21.15.98:0/2375417514 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57c805f5f0 con 0x7f57c80fb970 2024-04-22T21:02:30.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.826+0000 7f57c4ff9700 1 -- 172.21.15.98:0/2375417514 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f57b8011b60 con 0x7f57c80fb970 2024-04-22T21:02:30.823 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:02:30.823 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:00:19.700503Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:02:30.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.828+0000 7f57cdec2700 1 -- 172.21.15.98:0/2375417514 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f57b4036880 msgr2=0x7f57b4038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:30.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.828+0000 7f57cdec2700 1 --2- 172.21.15.98:0/2375417514 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f57b4036880 0x7f57b4038d30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009990 tx=0x7f57b0006e40 comp rx=0 tx=0).stop 2024-04-22T21:02:30.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.828+0000 7f57cdec2700 1 -- 172.21.15.98:0/2375417514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57c80fb970 msgr2=0x7f57c8065be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:30.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.828+0000 7f57cdec2700 1 --2- 172.21.15.98:0/2375417514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57c80fb970 0x7f57c8065be0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f57b80096b0 tx=0x7f57b8009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:30.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.828+0000 7f57cdec2700 1 -- 172.21.15.98:0/2375417514 shutdown_connections 2024-04-22T21:02:30.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.828+0000 7f57cdec2700 1 --2- 172.21.15.98:0/2375417514 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f57b4036880 0x7f57b4038d30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:30.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.828+0000 7f57cdec2700 1 --2- 172.21.15.98:0/2375417514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57c80fb970 0x7f57c8065be0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:30.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.828+0000 7f57cdec2700 1 -- 172.21.15.98:0/2375417514 >> 172.21.15.98:0/2375417514 conn(0x7f57c80f70d0 msgr2=0x7f57c8102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:30.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.828+0000 7f57cdec2700 1 -- 172.21.15.98:0/2375417514 shutdown_connections 2024-04-22T21:02:30.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:30.828+0000 7f57cdec2700 1 -- 172.21.15.98:0/2375417514 wait complete. 2024-04-22T21:02:30.826 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-22T21:02:31.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:31 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:31.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:31 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:31.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:31 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:31.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:31 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T21:02:31.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:31 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/2375417514' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:02:31.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:31 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:32.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:32 smithi038 ceph-mon[26374]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T21:02:32.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:32 smithi038 ceph-mon[26374]: mgrmap e14: smithi038.lxfgut(active, since 60s) 2024-04-22T21:02:32.457 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:02:32.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:02:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:32.752 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:34.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.082+0000 7f8e1ba1f700 1 -- 172.21.15.98:0/3338159282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e140fe180 msgr2=0x7f8e140fe590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:34.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.082+0000 7f8e1ba1f700 1 --2- 172.21.15.98:0/3338159282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e140fe180 0x7f8e140fe590 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c009c60 tx=0x7f8e0c004d10 comp rx=0 tx=0).stop 2024-04-22T21:02:34.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.082+0000 7f8e1ba1f700 1 -- 172.21.15.98:0/3338159282 shutdown_connections 2024-04-22T21:02:34.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.082+0000 7f8e1ba1f700 1 --2- 172.21.15.98:0/3338159282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e140fe180 0x7f8e140fe590 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:34.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.082+0000 7f8e1ba1f700 1 -- 172.21.15.98:0/3338159282 >> 172.21.15.98:0/3338159282 conn(0x7f8e140f6860 msgr2=0x7f8e140f8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:34.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.082+0000 7f8e1ba1f700 1 -- 172.21.15.98:0/3338159282 shutdown_connections 2024-04-22T21:02:34.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.082+0000 7f8e1ba1f700 1 -- 172.21.15.98:0/3338159282 wait complete. 2024-04-22T21:02:34.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.083+0000 7f8e1ba1f700 1 Processor -- start 2024-04-22T21:02:34.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.083+0000 7f8e1ba1f700 1 -- start start 2024-04-22T21:02:34.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.083+0000 7f8e1ba1f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e140fe180 0x7f8e1410d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:34.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.083+0000 7f8e1ba1f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1410d650 con 0x7f8e140fe180 2024-04-22T21:02:34.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.083+0000 7f8e197bb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e140fe180 0x7f8e1410d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:34.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.083+0000 7f8e197bb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e140fe180 0x7f8e1410d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:57464/0 (socket says 172.21.15.98:57464) 2024-04-22T21:02:34.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.083+0000 7f8e197bb700 1 -- 172.21.15.98:0/2749463796 learned_addr learned my addr 172.21.15.98:0/2749463796 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:02:34.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.084+0000 7f8e197bb700 1 -- 172.21.15.98:0/2749463796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c009940 con 0x7f8e140fe180 2024-04-22T21:02:34.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.084+0000 7f8e197bb700 1 --2- 172.21.15.98:0/2749463796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e140fe180 0x7f8e1410d110 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c004e60 tx=0x7f8e0c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:34.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.084+0000 7f8e09ffb700 1 -- 172.21.15.98:0/2749463796 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e0c012410 con 0x7f8e140fe180 2024-04-22T21:02:34.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.084+0000 7f8e09ffb700 1 -- 172.21.15.98:0/2749463796 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e0c012a40 con 0x7f8e140fe180 2024-04-22T21:02:34.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.084+0000 7f8e09ffb700 1 -- 172.21.15.98:0/2749463796 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e0c01ba40 con 0x7f8e140fe180 2024-04-22T21:02:34.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.084+0000 7f8e1ba1f700 1 -- 172.21.15.98:0/2749463796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1410d850 con 0x7f8e140fe180 2024-04-22T21:02:34.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.084+0000 7f8e1ba1f700 1 -- 172.21.15.98:0/2749463796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1410e3c0 con 0x7f8e140fe180 2024-04-22T21:02:34.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.085+0000 7f8e1ba1f700 1 -- 172.21.15.98:0/2749463796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e14102f90 con 0x7f8e140fe180 2024-04-22T21:02:34.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.085+0000 7f8e09ffb700 1 -- 172.21.15.98:0/2749463796 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 14) v1 ==== 43446+0+0 (secure 0 0 0) 0x7f8e0c019040 con 0x7f8e140fe180 2024-04-22T21:02:34.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.085+0000 7f8e09ffb700 1 --2- 172.21.15.98:0/2749463796 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f8e000368d0 0x7f8e00038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:34.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.085+0000 7f8e09ffb700 1 -- 172.21.15.98:0/2749463796 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8e0c049d60 con 0x7f8e140fe180 2024-04-22T21:02:34.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.085+0000 7f8e18fba700 1 -- 172.21.15.98:0/2749463796 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f8e000368d0 msgr2=0x7f8e00038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/989134174 2024-04-22T21:02:34.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.085+0000 7f8e18fba700 1 --2- 172.21.15.98:0/2749463796 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f8e000368d0 0x7f8e00038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T21:02:34.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.090+0000 7f8e09ffb700 1 -- 172.21.15.98:0/2749463796 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8e0c022b90 con 0x7f8e140fe180 2024-04-22T21:02:34.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.286+0000 7f8e18fba700 1 -- 172.21.15.98:0/2749463796 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f8e000368d0 msgr2=0x7f8e00038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/989134174 2024-04-22T21:02:34.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.286+0000 7f8e18fba700 1 --2- 172.21.15.98:0/2749463796 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f8e000368d0 0x7f8e00038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T21:02:34.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.352+0000 7f8e1ba1f700 1 -- 172.21.15.98:0/2749463796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e1410ee90 con 0x7f8e140fe180 2024-04-22T21:02:34.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.353+0000 7f8e09ffb700 1 -- 172.21.15.98:0/2749463796 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f8e0c017090 con 0x7f8e140fe180 2024-04-22T21:02:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:02:34.349 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:00:19.700503Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:02:34.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.355+0000 7f8e1ba1f700 1 -- 172.21.15.98:0/2749463796 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f8e000368d0 msgr2=0x7f8e00038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T21:02:34.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.355+0000 7f8e1ba1f700 1 --2- 172.21.15.98:0/2749463796 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f8e000368d0 0x7f8e00038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:34.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.355+0000 7f8e1ba1f700 1 -- 172.21.15.98:0/2749463796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e140fe180 msgr2=0x7f8e1410d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:34.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.355+0000 7f8e1ba1f700 1 --2- 172.21.15.98:0/2749463796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e140fe180 0x7f8e1410d110 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c004e60 tx=0x7f8e0c004d10 comp rx=0 tx=0).stop 2024-04-22T21:02:34.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.355+0000 7f8e1ba1f700 1 -- 172.21.15.98:0/2749463796 shutdown_connections 2024-04-22T21:02:34.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.355+0000 7f8e1ba1f700 1 --2- 172.21.15.98:0/2749463796 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f8e000368d0 0x7f8e00038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:34.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.355+0000 7f8e1ba1f700 1 --2- 172.21.15.98:0/2749463796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e140fe180 0x7f8e1410d110 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:34.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.355+0000 7f8e1ba1f700 1 -- 172.21.15.98:0/2749463796 >> 172.21.15.98:0/2749463796 conn(0x7f8e140f6860 msgr2=0x7f8e140f7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:34.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.355+0000 7f8e1ba1f700 1 -- 172.21.15.98:0/2749463796 shutdown_connections 2024-04-22T21:02:34.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:34.355+0000 7f8e1ba1f700 1 -- 172.21.15.98:0/2749463796 wait complete. 2024-04-22T21:02:34.352 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-22T21:02:34.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:34 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/2749463796' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:02:35.994 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:02:35.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:02:36.243 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:36.291 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:37.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.748+0000 7f03e361e700 1 -- 172.21.15.98:0/243328714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03dc0669f0 msgr2=0x7f03dc0fd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:37.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.748+0000 7f03e361e700 1 --2- 172.21.15.98:0/243328714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03dc0669f0 0x7f03dc0fd440 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f03d80048c0 tx=0x7f03d8009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:37.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.748+0000 7f03e361e700 1 -- 172.21.15.98:0/243328714 shutdown_connections 2024-04-22T21:02:37.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.748+0000 7f03e361e700 1 --2- 172.21.15.98:0/243328714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03dc0669f0 0x7f03dc0fd440 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:37.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.748+0000 7f03e361e700 1 -- 172.21.15.98:0/243328714 >> 172.21.15.98:0/243328714 conn(0x7f03dc0f70c0 msgr2=0x7f03dc0f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:37.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.748+0000 7f03e361e700 1 -- 172.21.15.98:0/243328714 shutdown_connections 2024-04-22T21:02:37.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.748+0000 7f03e361e700 1 -- 172.21.15.98:0/243328714 wait complete. 2024-04-22T21:02:37.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.748+0000 7f03e361e700 1 Processor -- start 2024-04-22T21:02:37.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.748+0000 7f03e361e700 1 -- start start 2024-04-22T21:02:37.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.749+0000 7f03e361e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03dc0669f0 0x7f03dc113730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:37.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.749+0000 7f03e361e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc113c70 con 0x7f03dc0669f0 2024-04-22T21:02:37.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.749+0000 7f03e13ba700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03dc0669f0 0x7f03dc113730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:37.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.749+0000 7f03e13ba700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03dc0669f0 0x7f03dc113730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:57480/0 (socket says 172.21.15.98:57480) 2024-04-22T21:02:37.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.749+0000 7f03e13ba700 1 -- 172.21.15.98:0/605793336 learned_addr learned my addr 172.21.15.98:0/605793336 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:02:37.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.749+0000 7f03e13ba700 1 -- 172.21.15.98:0/605793336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d8005040 con 0x7f03dc0669f0 2024-04-22T21:02:37.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.749+0000 7f03e13ba700 1 --2- 172.21.15.98:0/605793336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03dc0669f0 0x7f03dc113730 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009680 tx=0x7f03d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:37.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.749+0000 7f03d1ffb700 1 -- 172.21.15.98:0/605793336 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03d800bed0 con 0x7f03dc0669f0 2024-04-22T21:02:37.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.749+0000 7f03e361e700 1 -- 172.21.15.98:0/605793336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03dc113e70 con 0x7f03dc0669f0 2024-04-22T21:02:37.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.749+0000 7f03d1ffb700 1 -- 172.21.15.98:0/605793336 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03d800b5c0 con 0x7f03dc0669f0 2024-04-22T21:02:37.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.749+0000 7f03d1ffb700 1 -- 172.21.15.98:0/605793336 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03d801c5b0 con 0x7f03dc0669f0 2024-04-22T21:02:37.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.750+0000 7f03e361e700 1 -- 172.21.15.98:0/605793336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03dc114990 con 0x7f03dc0669f0 2024-04-22T21:02:37.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.750+0000 7f03d1ffb700 1 -- 172.21.15.98:0/605793336 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 14) v1 ==== 43446+0+0 (secure 0 0 0) 0x7f03d8013420 con 0x7f03dc0669f0 2024-04-22T21:02:37.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.750+0000 7f03e361e700 1 -- 172.21.15.98:0/605793336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03dc100d50 con 0x7f03dc0669f0 2024-04-22T21:02:37.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.751+0000 7f03d1ffb700 1 --2- 172.21.15.98:0/605793336 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f03c80368d0 0x7f03c8038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:37.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.751+0000 7f03d1ffb700 1 -- 172.21.15.98:0/605793336 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f03d8049a80 con 0x7f03dc0669f0 2024-04-22T21:02:37.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.751+0000 7f03e0bb9700 1 -- 172.21.15.98:0/605793336 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f03c80368d0 msgr2=0x7f03c8038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/989134174 2024-04-22T21:02:37.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.751+0000 7f03e0bb9700 1 --2- 172.21.15.98:0/605793336 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f03c80368d0 0x7f03c8038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T21:02:37.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.755+0000 7f03d1ffb700 1 -- 172.21.15.98:0/605793336 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f03d8013a80 con 0x7f03dc0669f0 2024-04-22T21:02:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.952+0000 7f03e0bb9700 1 -- 172.21.15.98:0/605793336 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f03c80368d0 msgr2=0x7f03c8038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/989134174 2024-04-22T21:02:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:37.952+0000 7f03e0bb9700 1 --2- 172.21.15.98:0/605793336 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f03c80368d0 0x7f03c8038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T21:02:38.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:38.013+0000 7f03e361e700 1 -- 172.21.15.98:0/605793336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03dc1153a0 con 0x7f03dc0669f0 2024-04-22T21:02:38.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:38.014+0000 7f03d1ffb700 1 -- 172.21.15.98:0/605793336 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f03d8011370 con 0x7f03dc0669f0 2024-04-22T21:02:38.010 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:02:38.010 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:00:19.700503Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:02:38.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:38.016+0000 7f03e361e700 1 -- 172.21.15.98:0/605793336 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f03c80368d0 msgr2=0x7f03c8038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T21:02:38.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:38.016+0000 7f03e361e700 1 --2- 172.21.15.98:0/605793336 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f03c80368d0 0x7f03c8038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:38.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:38.016+0000 7f03e361e700 1 -- 172.21.15.98:0/605793336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03dc0669f0 msgr2=0x7f03dc113730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:38.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:38.016+0000 7f03e361e700 1 --2- 172.21.15.98:0/605793336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03dc0669f0 0x7f03dc113730 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009680 tx=0x7f03d8009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:38.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:38.016+0000 7f03e361e700 1 -- 172.21.15.98:0/605793336 shutdown_connections 2024-04-22T21:02:38.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:38.016+0000 7f03e361e700 1 --2- 172.21.15.98:0/605793336 >> [v2:172.21.15.38:6800/989134174,v1:172.21.15.38:6801/989134174] conn(0x7f03c80368d0 0x7f03c8038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:38.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:38.016+0000 7f03e361e700 1 --2- 172.21.15.98:0/605793336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03dc0669f0 0x7f03dc113730 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:38.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:38.016+0000 7f03e361e700 1 -- 172.21.15.98:0/605793336 >> 172.21.15.98:0/605793336 conn(0x7f03dc0f70c0 msgr2=0x7f03dc10a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:38.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:38.016+0000 7f03e361e700 1 -- 172.21.15.98:0/605793336 shutdown_connections 2024-04-22T21:02:38.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:38.016+0000 7f03e361e700 1 -- 172.21.15.98:0/605793336 wait complete. 2024-04-22T21:02:38.013 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-22T21:02:38.082 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:38 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/605793336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:02:38.082 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:38 smithi038 ceph-mon[26374]: Active manager daemon smithi038.lxfgut restarted 2024-04-22T21:02:38.082 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:38 smithi038 ceph-mon[26374]: Activating manager daemon smithi038.lxfgut 2024-04-22T21:02:38.082 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:38 smithi038 ceph-mon[26374]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T21:02:38.082 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:38 smithi038 ceph-mon[26374]: mgrmap e15: smithi038.lxfgut(active, starting, since 0.00475133s) 2024-04-22T21:02:38.082 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2024-04-22T21:02:38.082 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr metadata", "who": "smithi038.lxfgut", "id": "smithi038.lxfgut"}]: dispatch 2024-04-22T21:02:38.082 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T21:02:38.082 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T21:02:38.083 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T21:02:38.083 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:38 smithi038 ceph-mon[26374]: Manager daemon smithi038.lxfgut is now available 2024-04-22T21:02:39.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:39 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:02:39.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:39 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:02:39.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:39 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:39.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:39 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:02:39.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:39 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.lxfgut/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:02:39.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:39 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.lxfgut/trash_purge_schedule"}]: dispatch 2024-04-22T21:02:39.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:39 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:39.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:39 smithi038 ceph-mon[26374]: mgrmap e16: smithi038.lxfgut(active, since 1.00844s) 2024-04-22T21:02:39.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:39 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:39.643 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:02:39.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:02:39.891 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:40.668 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:40 smithi038 ceph-mon[26374]: [22/Apr/2024:21:02:39] ENGINE Bus STARTING 2024-04-22T21:02:40.669 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:40 smithi038 ceph-mon[26374]: [22/Apr/2024:21:02:39] ENGINE Serving on https://172.21.15.38:7150 2024-04-22T21:02:40.669 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:40 smithi038 ceph-mon[26374]: [22/Apr/2024:21:02:39] ENGINE Bus STARTED 2024-04-22T21:02:40.669 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:40 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:40.669 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:40 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:41.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.418+0000 7f36ec052700 1 -- 172.21.15.98:0/1928108724 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36e40fca60 msgr2=0x7f36e40fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.418+0000 7f36ec052700 1 --2- 172.21.15.98:0/1928108724 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36e40fca60 0x7f36e40fce70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009c60 tx=0x7f36dc004d10 comp rx=0 tx=0).stop 2024-04-22T21:02:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.418+0000 7f36ec052700 1 -- 172.21.15.98:0/1928108724 shutdown_connections 2024-04-22T21:02:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.418+0000 7f36ec052700 1 --2- 172.21.15.98:0/1928108724 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36e40fca60 0x7f36e40fce70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.418+0000 7f36ec052700 1 -- 172.21.15.98:0/1928108724 >> 172.21.15.98:0/1928108724 conn(0x7f36e40f81c0 msgr2=0x7f36e40fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.418+0000 7f36ec052700 1 -- 172.21.15.98:0/1928108724 shutdown_connections 2024-04-22T21:02:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.418+0000 7f36ec052700 1 -- 172.21.15.98:0/1928108724 wait complete. 2024-04-22T21:02:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.418+0000 7f36ec052700 1 Processor -- start 2024-04-22T21:02:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.418+0000 7f36ec052700 1 -- start start 2024-04-22T21:02:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.418+0000 7f36ec052700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36e40fca60 0x7f36e4071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.418+0000 7f36ec052700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e40722a0 con 0x7f36e40fca60 2024-04-22T21:02:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.419+0000 7f36e9dee700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36e40fca60 0x7f36e4071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.419+0000 7f36e9dee700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36e40fca60 0x7f36e4071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:35408/0 (socket says 172.21.15.98:35408) 2024-04-22T21:02:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.419+0000 7f36e9dee700 1 -- 172.21.15.98:0/2295705493 learned_addr learned my addr 172.21.15.98:0/2295705493 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:02:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.419+0000 7f36e9dee700 1 -- 172.21.15.98:0/2295705493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36dc009940 con 0x7f36e40fca60 2024-04-22T21:02:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.419+0000 7f36e9dee700 1 --2- 172.21.15.98:0/2295705493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36e40fca60 0x7f36e4071d60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009500 tx=0x7f36dc004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.419+0000 7f36da7fc700 1 -- 172.21.15.98:0/2295705493 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36dc007c80 con 0x7f36e40fca60 2024-04-22T21:02:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.419+0000 7f36ec052700 1 -- 172.21.15.98:0/2295705493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e4070370 con 0x7f36e40fca60 2024-04-22T21:02:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.419+0000 7f36da7fc700 1 -- 172.21.15.98:0/2295705493 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36dc01d910 con 0x7f36e40fca60 2024-04-22T21:02:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.419+0000 7f36da7fc700 1 -- 172.21.15.98:0/2295705493 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36dc0129d0 con 0x7f36e40fca60 2024-04-22T21:02:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.419+0000 7f36ec052700 1 -- 172.21.15.98:0/2295705493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e4070860 con 0x7f36e40fca60 2024-04-22T21:02:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.420+0000 7f36da7fc700 1 -- 172.21.15.98:0/2295705493 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7f36dc019040 con 0x7f36e40fca60 2024-04-22T21:02:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.420+0000 7f36da7fc700 1 --2- 172.21.15.98:0/2295705493 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f36d0036970 0x7f36d0038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.420+0000 7f36ec052700 1 -- 172.21.15.98:0/2295705493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e405f5f0 con 0x7f36e40fca60 2024-04-22T21:02:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.420+0000 7f36da7fc700 1 -- 172.21.15.98:0/2295705493 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f36dc04a610 con 0x7f36e40fca60 2024-04-22T21:02:41.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.420+0000 7f36e95ed700 1 --2- 172.21.15.98:0/2295705493 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f36d0036970 0x7f36d0038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:41.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.421+0000 7f36e95ed700 1 --2- 172.21.15.98:0/2295705493 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f36d0036970 0x7f36d0038e20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f36d4009990 tx=0x7f36d4006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:41.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.425+0000 7f36da7fc700 1 -- 172.21.15.98:0/2295705493 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f36dc021080 con 0x7f36e40fca60 2024-04-22T21:02:41.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.695+0000 7f36ec052700 1 -- 172.21.15.98:0/2295705493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36e404bed0 con 0x7f36e40fca60 2024-04-22T21:02:41.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.695+0000 7f36da7fc700 1 -- 172.21.15.98:0/2295705493 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f36dc017030 con 0x7f36e40fca60 2024-04-22T21:02:41.692 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:02:41.692 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:00:19.700503Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:02:41.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.697+0000 7f36ec052700 1 -- 172.21.15.98:0/2295705493 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f36d0036970 msgr2=0x7f36d0038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:41.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.697+0000 7f36ec052700 1 --2- 172.21.15.98:0/2295705493 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f36d0036970 0x7f36d0038e20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f36d4009990 tx=0x7f36d4006e40 comp rx=0 tx=0).stop 2024-04-22T21:02:41.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.697+0000 7f36ec052700 1 -- 172.21.15.98:0/2295705493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36e40fca60 msgr2=0x7f36e4071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:41.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.697+0000 7f36ec052700 1 --2- 172.21.15.98:0/2295705493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36e40fca60 0x7f36e4071d60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009500 tx=0x7f36dc004d10 comp rx=0 tx=0).stop 2024-04-22T21:02:41.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.697+0000 7f36ec052700 1 -- 172.21.15.98:0/2295705493 shutdown_connections 2024-04-22T21:02:41.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.697+0000 7f36ec052700 1 --2- 172.21.15.98:0/2295705493 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f36d0036970 0x7f36d0038e20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:41.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.697+0000 7f36ec052700 1 --2- 172.21.15.98:0/2295705493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36e40fca60 0x7f36e4071d60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:41.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.697+0000 7f36ec052700 1 -- 172.21.15.98:0/2295705493 >> 172.21.15.98:0/2295705493 conn(0x7f36e40f81c0 msgr2=0x7f36e4103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:41.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.698+0000 7f36ec052700 1 -- 172.21.15.98:0/2295705493 shutdown_connections 2024-04-22T21:02:41.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:41.698+0000 7f36ec052700 1 -- 172.21.15.98:0/2295705493 wait complete. 2024-04-22T21:02:41.695 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-22T21:02:41.949 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:41 smithi038 ceph-mon[26374]: mgrmap e17: smithi038.lxfgut(active, since 2s) 2024-04-22T21:02:41.950 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:41 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/2295705493' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:02:43.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:43 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:43.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:43 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:43.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:43 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:43.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:43 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:43.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:43 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:02:43.909 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:02:43.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:02:44.600 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:44.660 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T21:02:45.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:45.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:45.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:45.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:45.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:02:45.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:02:45.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:02:45.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-22T21:02:45.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: Updating smithi098:/etc/ceph/ceph.conf 2024-04-22T21:02:45.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: Updating smithi098:/var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T21:02:45.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: Updating smithi038:/var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T21:02:45.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:02:45.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:02:45.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: Updating smithi098:/var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T21:02:45.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: Updating smithi038:/var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T21:02:45.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:45.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:45.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:45.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:45.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:45.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi098", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T21:02:45.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi098", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-22T21:02:45.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:02:46.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.344+0000 7f467dacc700 1 -- 172.21.15.98:0/1434539421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46781019c0 msgr2=0x7f4678101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:46.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.344+0000 7f467dacc700 1 --2- 172.21.15.98:0/1434539421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46781019c0 0x7f4678101d90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f46680048c0 tx=0x7f4668009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:46.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.344+0000 7f467dacc700 1 -- 172.21.15.98:0/1434539421 shutdown_connections 2024-04-22T21:02:46.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.344+0000 7f467dacc700 1 --2- 172.21.15.98:0/1434539421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46781019c0 0x7f4678101d90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:46.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.344+0000 7f467dacc700 1 -- 172.21.15.98:0/1434539421 >> 172.21.15.98:0/1434539421 conn(0x7f46780f7520 msgr2=0x7f46780f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:46.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.344+0000 7f467dacc700 1 -- 172.21.15.98:0/1434539421 shutdown_connections 2024-04-22T21:02:46.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.344+0000 7f467dacc700 1 -- 172.21.15.98:0/1434539421 wait complete. 2024-04-22T21:02:46.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.344+0000 7f467dacc700 1 Processor -- start 2024-04-22T21:02:46.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.344+0000 7f467dacc700 1 -- start start 2024-04-22T21:02:46.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.344+0000 7f467dacc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46781019c0 0x7f46780f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:46.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.344+0000 7f467dacc700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46780f7f00 con 0x7f46781019c0 2024-04-22T21:02:46.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.345+0000 7f46777fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46781019c0 0x7f46780f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:46.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.345+0000 7f46777fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46781019c0 0x7f46780f79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:35428/0 (socket says 172.21.15.98:35428) 2024-04-22T21:02:46.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.345+0000 7f46777fe700 1 -- 172.21.15.98:0/202986320 learned_addr learned my addr 172.21.15.98:0/202986320 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:02:46.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.345+0000 7f46777fe700 1 -- 172.21.15.98:0/202986320 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4668005040 con 0x7f46781019c0 2024-04-22T21:02:46.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.345+0000 7f46777fe700 1 --2- 172.21.15.98:0/202986320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46781019c0 0x7f46780f79c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f466800b820 tx=0x7f4668009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:46.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.345+0000 7f4674ff9700 1 -- 172.21.15.98:0/202986320 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46680093c0 con 0x7f46781019c0 2024-04-22T21:02:46.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.345+0000 7f467dacc700 1 -- 172.21.15.98:0/202986320 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46780f8100 con 0x7f46781019c0 2024-04-22T21:02:46.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.345+0000 7f4674ff9700 1 -- 172.21.15.98:0/202986320 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4668013420 con 0x7f46781019c0 2024-04-22T21:02:46.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.345+0000 7f4674ff9700 1 -- 172.21.15.98:0/202986320 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f466801c5b0 con 0x7f46781019c0 2024-04-22T21:02:46.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.345+0000 7f467dacc700 1 -- 172.21.15.98:0/202986320 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46780f85f0 con 0x7f46781019c0 2024-04-22T21:02:46.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.346+0000 7f467dacc700 1 -- 172.21.15.98:0/202986320 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46780f8830 con 0x7f46781019c0 2024-04-22T21:02:46.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.346+0000 7f4674ff9700 1 -- 172.21.15.98:0/202986320 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7f4668013a90 con 0x7f46781019c0 2024-04-22T21:02:46.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.346+0000 7f4674ff9700 1 --2- 172.21.15.98:0/202986320 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f4664036970 0x7f4664038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:46.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.346+0000 7f4674ff9700 1 -- 172.21.15.98:0/202986320 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f4668049a70 con 0x7f46781019c0 2024-04-22T21:02:46.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.347+0000 7f466ffff700 1 --2- 172.21.15.98:0/202986320 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f4664036970 0x7f4664038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:46.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.347+0000 7f466ffff700 1 --2- 172.21.15.98:0/202986320 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f4664036970 0x7f4664038e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4660009990 tx=0x7f4660006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.352+0000 7f4674ff9700 1 -- 172.21.15.98:0/202986320 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f46780f8830 con 0x7f46781019c0 2024-04-22T21:02:46.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.621+0000 7f467dacc700 1 -- 172.21.15.98:0/202986320 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46780f8830 con 0x7f46781019c0 2024-04-22T21:02:46.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.621+0000 7f4674ff9700 1 -- 172.21.15.98:0/202986320 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f46780f8830 con 0x7f46781019c0 2024-04-22T21:02:46.618 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:02:46.618 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:00:19.700503Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:02:46.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.623+0000 7f467dacc700 1 -- 172.21.15.98:0/202986320 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f4664036970 msgr2=0x7f4664038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:46.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.623+0000 7f467dacc700 1 --2- 172.21.15.98:0/202986320 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f4664036970 0x7f4664038e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4660009990 tx=0x7f4660006e40 comp rx=0 tx=0).stop 2024-04-22T21:02:46.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.624+0000 7f467dacc700 1 -- 172.21.15.98:0/202986320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46781019c0 msgr2=0x7f46780f79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:46.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.624+0000 7f467dacc700 1 --2- 172.21.15.98:0/202986320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46781019c0 0x7f46780f79c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f466800b820 tx=0x7f4668009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:46.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.624+0000 7f467dacc700 1 -- 172.21.15.98:0/202986320 shutdown_connections 2024-04-22T21:02:46.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.624+0000 7f467dacc700 1 --2- 172.21.15.98:0/202986320 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f4664036970 0x7f4664038e20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:46.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.624+0000 7f467dacc700 1 --2- 172.21.15.98:0/202986320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46781019c0 0x7f46780f79c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:46.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.624+0000 7f467dacc700 1 -- 172.21.15.98:0/202986320 >> 172.21.15.98:0/202986320 conn(0x7f46780f7520 msgr2=0x7f4678065ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:46.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.624+0000 7f467dacc700 1 -- 172.21.15.98:0/202986320 shutdown_connections 2024-04-22T21:02:46.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:46.624+0000 7f467dacc700 1 -- 172.21.15.98:0/202986320 wait complete. 2024-04-22T21:02:46.621 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-22T21:02:46.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:46 smithi038 ceph-mon[26374]: Deploying daemon crash.smithi098 on smithi098 2024-04-22T21:02:47.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:47 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/202986320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:02:48.299 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:02:48.300 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:02:48.632 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T21:02:50.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.147+0000 7f6f2cdc8700 1 -- 172.21.15.98:0/1726661383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f28102ae0 msgr2=0x7f6f28102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:50.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.147+0000 7f6f2cdc8700 1 --2- 172.21.15.98:0/1726661383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f28102ae0 0x7f6f28102eb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6f180048c0 tx=0x7f6f18009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:50.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.147+0000 7f6f2cdc8700 1 -- 172.21.15.98:0/1726661383 shutdown_connections 2024-04-22T21:02:50.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.147+0000 7f6f2cdc8700 1 --2- 172.21.15.98:0/1726661383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f28102ae0 0x7f6f28102eb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:50.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.147+0000 7f6f2cdc8700 1 -- 172.21.15.98:0/1726661383 >> 172.21.15.98:0/1726661383 conn(0x7f6f280f85e0 msgr2=0x7f6f280fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:50.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.147+0000 7f6f2cdc8700 1 -- 172.21.15.98:0/1726661383 shutdown_connections 2024-04-22T21:02:50.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.147+0000 7f6f2cdc8700 1 -- 172.21.15.98:0/1726661383 wait complete. 2024-04-22T21:02:50.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.148+0000 7f6f2cdc8700 1 Processor -- start 2024-04-22T21:02:50.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.148+0000 7f6f2cdc8700 1 -- start start 2024-04-22T21:02:50.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.148+0000 7f6f2cdc8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f28102ae0 0x7f6f280702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:50.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.148+0000 7f6f2cdc8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f280722c0 con 0x7f6f28102ae0 2024-04-22T21:02:50.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.148+0000 7f6f2659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f28102ae0 0x7f6f280702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:50.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.148+0000 7f6f2659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f28102ae0 0x7f6f280702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:37084/0 (socket says 172.21.15.98:37084) 2024-04-22T21:02:50.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.148+0000 7f6f2659c700 1 -- 172.21.15.98:0/2244039159 learned_addr learned my addr 172.21.15.98:0/2244039159 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:02:50.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.148+0000 7f6f2659c700 1 -- 172.21.15.98:0/2244039159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f18005040 con 0x7f6f28102ae0 2024-04-22T21:02:50.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.149+0000 7f6f2659c700 1 --2- 172.21.15.98:0/2244039159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f28102ae0 0x7f6f280702c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800b820 tx=0x7f6f18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:50.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.149+0000 7f6f16ffd700 1 -- 172.21.15.98:0/2244039159 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f180093c0 con 0x7f6f28102ae0 2024-04-22T21:02:50.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.149+0000 7f6f2cdc8700 1 -- 172.21.15.98:0/2244039159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f28070860 con 0x7f6f28102ae0 2024-04-22T21:02:50.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.149+0000 7f6f16ffd700 1 -- 172.21.15.98:0/2244039159 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f18013420 con 0x7f6f28102ae0 2024-04-22T21:02:50.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.149+0000 7f6f16ffd700 1 -- 172.21.15.98:0/2244039159 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f1801c5b0 con 0x7f6f28102ae0 2024-04-22T21:02:50.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.149+0000 7f6f2cdc8700 1 -- 172.21.15.98:0/2244039159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f28070cd0 con 0x7f6f28102ae0 2024-04-22T21:02:50.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.150+0000 7f6f2cdc8700 1 -- 172.21.15.98:0/2244039159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f280fe1e0 con 0x7f6f28102ae0 2024-04-22T21:02:50.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.150+0000 7f6f16ffd700 1 -- 172.21.15.98:0/2244039159 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7f6f18013a90 con 0x7f6f28102ae0 2024-04-22T21:02:50.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.150+0000 7f6f16ffd700 1 --2- 172.21.15.98:0/2244039159 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6f10036970 0x7f6f10038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:50.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.150+0000 7f6f16ffd700 1 -- 172.21.15.98:0/2244039159 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f6f18049b10 con 0x7f6f28102ae0 2024-04-22T21:02:50.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.150+0000 7f6f25d9b700 1 --2- 172.21.15.98:0/2244039159 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6f10036970 0x7f6f10038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:50.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.151+0000 7f6f25d9b700 1 --2- 172.21.15.98:0/2244039159 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6f10036970 0x7f6f10038e20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c009990 tx=0x7f6f1c006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:50.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.155+0000 7f6f16ffd700 1 -- 172.21.15.98:0/2244039159 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6f180098a0 con 0x7f6f28102ae0 2024-04-22T21:02:50.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.433+0000 7f6f2cdc8700 1 -- 172.21.15.98:0/2244039159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f2804bf30 con 0x7f6f28102ae0 2024-04-22T21:02:50.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.433+0000 7f6f16ffd700 1 -- 172.21.15.98:0/2244039159 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f6f18011370 con 0x7f6f28102ae0 2024-04-22T21:02:50.430 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:02:50.430 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:00:19.700503Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:02:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.436+0000 7f6f2cdc8700 1 -- 172.21.15.98:0/2244039159 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6f10036970 msgr2=0x7f6f10038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.436+0000 7f6f2cdc8700 1 --2- 172.21.15.98:0/2244039159 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6f10036970 0x7f6f10038e20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c009990 tx=0x7f6f1c006e40 comp rx=0 tx=0).stop 2024-04-22T21:02:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.436+0000 7f6f2cdc8700 1 -- 172.21.15.98:0/2244039159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f28102ae0 msgr2=0x7f6f280702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.436+0000 7f6f2cdc8700 1 --2- 172.21.15.98:0/2244039159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f28102ae0 0x7f6f280702c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800b820 tx=0x7f6f18009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.436+0000 7f6f2cdc8700 1 -- 172.21.15.98:0/2244039159 shutdown_connections 2024-04-22T21:02:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.436+0000 7f6f2cdc8700 1 --2- 172.21.15.98:0/2244039159 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6f10036970 0x7f6f10038e20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.436+0000 7f6f2cdc8700 1 --2- 172.21.15.98:0/2244039159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f28102ae0 0x7f6f280702c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.436+0000 7f6f2cdc8700 1 -- 172.21.15.98:0/2244039159 >> 172.21.15.98:0/2244039159 conn(0x7f6f280f85e0 msgr2=0x7f6f280f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.436+0000 7f6f2cdc8700 1 -- 172.21.15.98:0/2244039159 shutdown_connections 2024-04-22T21:02:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:50.436+0000 7f6f2cdc8700 1 -- 172.21.15.98:0/2244039159 wait complete. 2024-04-22T21:02:50.433 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-22T21:02:50.864 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:50 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/2244039159' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:02:52.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:51 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:52.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:51 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:52.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:51 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:52.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:51 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:52.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:51 smithi038 ceph-mon[26374]: Deploying daemon node-exporter.smithi098 on smithi098 2024-04-22T21:02:52.183 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:02:52.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:02:52.434 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T21:02:54.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:54 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:54.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:54 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:02:54.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.943+0000 7fc1a921f700 1 -- 172.21.15.98:0/3026300792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1a4102b00 msgr2=0x7fc1a4102ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.943+0000 7fc1a921f700 1 --2- 172.21.15.98:0/3026300792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1a4102b00 0x7fc1a4102ed0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc1940048c0 tx=0x7fc194009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.943+0000 7fc1a921f700 1 -- 172.21.15.98:0/3026300792 shutdown_connections 2024-04-22T21:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.943+0000 7fc1a921f700 1 --2- 172.21.15.98:0/3026300792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1a4102b00 0x7fc1a4102ed0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.943+0000 7fc1a921f700 1 -- 172.21.15.98:0/3026300792 >> 172.21.15.98:0/3026300792 conn(0x7fc1a40f8600 msgr2=0x7fc1a40faa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.943+0000 7fc1a921f700 1 -- 172.21.15.98:0/3026300792 shutdown_connections 2024-04-22T21:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.943+0000 7fc1a921f700 1 -- 172.21.15.98:0/3026300792 wait complete. 2024-04-22T21:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.944+0000 7fc1a921f700 1 Processor -- start 2024-04-22T21:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.944+0000 7fc1a921f700 1 -- start start 2024-04-22T21:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.944+0000 7fc1a921f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1a4102b00 0x7fc1a4113180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.944+0000 7fc1a921f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a41136c0 con 0x7fc1a4102b00 2024-04-22T21:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.944+0000 7fc1a2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1a4102b00 0x7fc1a4113180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.944+0000 7fc1a2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1a4102b00 0x7fc1a4113180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:37096/0 (socket says 172.21.15.98:37096) 2024-04-22T21:02:54.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.944+0000 7fc1a2d9d700 1 -- 172.21.15.98:0/439004432 learned_addr learned my addr 172.21.15.98:0/439004432 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.945+0000 7fc1a2d9d700 1 -- 172.21.15.98:0/439004432 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc194005040 con 0x7fc1a4102b00 2024-04-22T21:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.945+0000 7fc1a2d9d700 1 --2- 172.21.15.98:0/439004432 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1a4102b00 0x7fc1a4113180 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc19400b820 tx=0x7fc194009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.945+0000 7fc18b7fe700 1 -- 172.21.15.98:0/439004432 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1940093c0 con 0x7fc1a4102b00 2024-04-22T21:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.945+0000 7fc18b7fe700 1 -- 172.21.15.98:0/439004432 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc194013420 con 0x7fc1a4102b00 2024-04-22T21:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.945+0000 7fc1a921f700 1 -- 172.21.15.98:0/439004432 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a41138c0 con 0x7fc1a4102b00 2024-04-22T21:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.945+0000 7fc18b7fe700 1 -- 172.21.15.98:0/439004432 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc19401c5b0 con 0x7fc1a4102b00 2024-04-22T21:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.945+0000 7fc1a921f700 1 -- 172.21.15.98:0/439004432 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a4113db0 con 0x7fc1a4102b00 2024-04-22T21:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.946+0000 7fc18b7fe700 1 -- 172.21.15.98:0/439004432 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7fc194009520 con 0x7fc1a4102b00 2024-04-22T21:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.946+0000 7fc1a921f700 1 -- 172.21.15.98:0/439004432 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a40fe200 con 0x7fc1a4102b00 2024-04-22T21:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.946+0000 7fc18b7fe700 1 --2- 172.21.15.98:0/439004432 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc18c036970 0x7fc18c038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.946+0000 7fc18b7fe700 1 -- 172.21.15.98:0/439004432 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fc194049aa0 con 0x7fc1a4102b00 2024-04-22T21:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.946+0000 7fc1a259c700 1 --2- 172.21.15.98:0/439004432 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc18c036970 0x7fc18c038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.947+0000 7fc1a259c700 1 --2- 172.21.15.98:0/439004432 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc18c036970 0x7fc18c038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc198009990 tx=0x7fc198006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:02:54.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:54.952+0000 7fc18b7fe700 1 -- 172.21.15.98:0/439004432 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fc194013a90 con 0x7fc1a4102b00 2024-04-22T21:02:55.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:55.218+0000 7fc1a921f700 1 -- 172.21.15.98:0/439004432 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1a405f5f0 con 0x7fc1a4102b00 2024-04-22T21:02:55.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:55.218+0000 7fc18b7fe700 1 -- 172.21.15.98:0/439004432 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fc194011370 con 0x7fc1a4102b00 2024-04-22T21:02:55.216 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:02:55.216 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:00:19.700503Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:02:55.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:55.220+0000 7fc1a921f700 1 -- 172.21.15.98:0/439004432 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc18c036970 msgr2=0x7fc18c038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:55.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:55.220+0000 7fc1a921f700 1 --2- 172.21.15.98:0/439004432 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc18c036970 0x7fc18c038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc198009990 tx=0x7fc198006e40 comp rx=0 tx=0).stop 2024-04-22T21:02:55.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:55.220+0000 7fc1a921f700 1 -- 172.21.15.98:0/439004432 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1a4102b00 msgr2=0x7fc1a4113180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:02:55.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:55.220+0000 7fc1a921f700 1 --2- 172.21.15.98:0/439004432 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1a4102b00 0x7fc1a4113180 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc19400b820 tx=0x7fc194009e30 comp rx=0 tx=0).stop 2024-04-22T21:02:55.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:55.220+0000 7fc1a921f700 1 -- 172.21.15.98:0/439004432 shutdown_connections 2024-04-22T21:02:55.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:55.220+0000 7fc1a921f700 1 --2- 172.21.15.98:0/439004432 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc18c036970 0x7fc18c038e20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:55.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:55.220+0000 7fc1a921f700 1 --2- 172.21.15.98:0/439004432 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1a4102b00 0x7fc1a4113180 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:02:55.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:55.220+0000 7fc1a921f700 1 -- 172.21.15.98:0/439004432 >> 172.21.15.98:0/439004432 conn(0x7fc1a40f8600 msgr2=0x7fc1a40f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:02:55.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:55.220+0000 7fc1a921f700 1 -- 172.21.15.98:0/439004432 shutdown_connections 2024-04-22T21:02:55.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:02:55.220+0000 7fc1a921f700 1 -- 172.21.15.98:0/439004432 wait complete. 2024-04-22T21:02:55.218 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-22T21:02:55.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:55 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/439004432' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:02:56.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:56 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:56.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:56 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:56.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:56 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:56.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:56 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:56.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:56 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi098.azqohf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:02:56.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:56 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi098.azqohf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T21:02:56.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:56 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:02:56.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:56 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:02:56.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:56 smithi038 ceph-mon[26374]: Deploying daemon mgr.smithi098.azqohf on smithi098 2024-04-22T21:02:57.001 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:02:57.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:02:57.266 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T21:02:59.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:59 smithi038 ceph-mon[26374]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:02:59.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:59 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:59.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:59 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:59.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:59 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:59.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:59 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:59.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:59 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:02:59.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:59 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:02:59.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:02:59 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:00.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.006+0000 7f1e16121700 1 -- 172.21.15.98:0/839487227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e10102b00 msgr2=0x7f1e10102ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:00.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.006+0000 7f1e16121700 1 --2- 172.21.15.98:0/839487227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e10102b00 0x7f1e10102ed0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1e000048c0 tx=0x7f1e00009e30 comp rx=0 tx=0).stop 2024-04-22T21:03:00.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.006+0000 7f1e16121700 1 -- 172.21.15.98:0/839487227 shutdown_connections 2024-04-22T21:03:00.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.006+0000 7f1e16121700 1 --2- 172.21.15.98:0/839487227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e10102b00 0x7f1e10102ed0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.006+0000 7f1e16121700 1 -- 172.21.15.98:0/839487227 >> 172.21.15.98:0/839487227 conn(0x7f1e100f8620 msgr2=0x7f1e100faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:03:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.006+0000 7f1e16121700 1 -- 172.21.15.98:0/839487227 shutdown_connections 2024-04-22T21:03:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.006+0000 7f1e16121700 1 -- 172.21.15.98:0/839487227 wait complete. 2024-04-22T21:03:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.006+0000 7f1e16121700 1 Processor -- start 2024-04-22T21:03:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.006+0000 7f1e16121700 1 -- start start 2024-04-22T21:03:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.007+0000 7f1e16121700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e10102b00 0x7f1e10113130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:03:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.007+0000 7f1e16121700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e10113670 con 0x7f1e10102b00 2024-04-22T21:03:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.007+0000 7f1e0f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e10102b00 0x7f1e10113130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:03:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.007+0000 7f1e0f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e10102b00 0x7f1e10113130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:42482/0 (socket says 172.21.15.98:42482) 2024-04-22T21:03:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.007+0000 7f1e0f7fe700 1 -- 172.21.15.98:0/2892043602 learned_addr learned my addr 172.21.15.98:0/2892043602 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:03:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.007+0000 7f1e0f7fe700 1 -- 172.21.15.98:0/2892043602 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e00005040 con 0x7f1e10102b00 2024-04-22T21:03:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.007+0000 7f1e0f7fe700 1 --2- 172.21.15.98:0/2892043602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e10102b00 0x7f1e10113130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000b820 tx=0x7f1e00009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:03:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.008+0000 7f1df7fff700 1 -- 172.21.15.98:0/2892043602 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e000093c0 con 0x7f1e10102b00 2024-04-22T21:03:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.008+0000 7f1df7fff700 1 -- 172.21.15.98:0/2892043602 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e00013420 con 0x7f1e10102b00 2024-04-22T21:03:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.008+0000 7f1e16121700 1 -- 172.21.15.98:0/2892043602 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e10113870 con 0x7f1e10102b00 2024-04-22T21:03:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.008+0000 7f1df7fff700 1 -- 172.21.15.98:0/2892043602 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e0001c5b0 con 0x7f1e10102b00 2024-04-22T21:03:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.008+0000 7f1e16121700 1 -- 172.21.15.98:0/2892043602 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e10113d60 con 0x7f1e10102b00 2024-04-22T21:03:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.008+0000 7f1df7fff700 1 -- 172.21.15.98:0/2892043602 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7f1e00009520 con 0x7f1e10102b00 2024-04-22T21:03:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.009+0000 7f1df7fff700 1 --2- 172.21.15.98:0/2892043602 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f1df8036970 0x7f1df8038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:03:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.009+0000 7f1df7fff700 1 -- 172.21.15.98:0/2892043602 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f1e00049a60 con 0x7f1e10102b00 2024-04-22T21:03:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.009+0000 7f1e0effd700 1 --2- 172.21.15.98:0/2892043602 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f1df8036970 0x7f1df8038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:03:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.009+0000 7f1e16121700 1 -- 172.21.15.98:0/2892043602 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e100fe200 con 0x7f1e10102b00 2024-04-22T21:03:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.009+0000 7f1e0effd700 1 --2- 172.21.15.98:0/2892043602 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f1df8036970 0x7f1df8038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1e04009990 tx=0x7f1e04006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:03:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.014+0000 7f1df7fff700 1 -- 172.21.15.98:0/2892043602 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1e00013a90 con 0x7f1e10102b00 2024-04-22T21:03:00.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.321+0000 7f1e16121700 1 -- 172.21.15.98:0/2892043602 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e1005f5f0 con 0x7f1e10102b00 2024-04-22T21:03:00.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.322+0000 7f1df7fff700 1 -- 172.21.15.98:0/2892043602 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f1e00034090 con 0x7f1e10102b00 2024-04-22T21:03:00.318 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:03:00.319 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:00:19.700503Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:03:00.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.324+0000 7f1e16121700 1 -- 172.21.15.98:0/2892043602 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f1df8036970 msgr2=0x7f1df8038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:00.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.324+0000 7f1e16121700 1 --2- 172.21.15.98:0/2892043602 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f1df8036970 0x7f1df8038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1e04009990 tx=0x7f1e04006e40 comp rx=0 tx=0).stop 2024-04-22T21:03:00.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.324+0000 7f1e16121700 1 -- 172.21.15.98:0/2892043602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e10102b00 msgr2=0x7f1e10113130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:00.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.324+0000 7f1e16121700 1 --2- 172.21.15.98:0/2892043602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e10102b00 0x7f1e10113130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000b820 tx=0x7f1e00009e30 comp rx=0 tx=0).stop 2024-04-22T21:03:00.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.324+0000 7f1e16121700 1 -- 172.21.15.98:0/2892043602 shutdown_connections 2024-04-22T21:03:00.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.324+0000 7f1e16121700 1 --2- 172.21.15.98:0/2892043602 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f1df8036970 0x7f1df8038e20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:00.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.324+0000 7f1e16121700 1 --2- 172.21.15.98:0/2892043602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e10102b00 0x7f1e10113130 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:00.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.324+0000 7f1e16121700 1 -- 172.21.15.98:0/2892043602 >> 172.21.15.98:0/2892043602 conn(0x7f1e100f8620 msgr2=0x7f1e100f9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:03:00.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.324+0000 7f1e16121700 1 -- 172.21.15.98:0/2892043602 shutdown_connections 2024-04-22T21:03:00.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:00.324+0000 7f1e16121700 1 -- 172.21.15.98:0/2892043602 wait complete. 2024-04-22T21:03:00.321 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-22T21:03:00.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:00 smithi038 ceph-mon[26374]: Deploying daemon mon.smithi098 on smithi098 2024-04-22T21:03:01.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:01 smithi038 ceph-mon[26374]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:01.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:01 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/2892043602' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:02.010 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:03:02.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:03:02.286 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi098/config 2024-04-22T21:03:03.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:03 smithi038 ceph-mon[26374]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:03.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:03 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:03.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.842+0000 7fbaba612700 1 -- 172.21.15.98:0/1594998591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab4102ba0 msgr2=0x7fbab4102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:03.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.842+0000 7fbaba612700 1 --2- 172.21.15.98:0/1594998591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab4102ba0 0x7fbab4102f70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbaa40048c0 tx=0x7fbaa4009e30 comp rx=0 tx=0).stop 2024-04-22T21:03:03.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.842+0000 7fbaba612700 1 -- 172.21.15.98:0/1594998591 shutdown_connections 2024-04-22T21:03:03.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.842+0000 7fbaba612700 1 --2- 172.21.15.98:0/1594998591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab4102ba0 0x7fbab4102f70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:03.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.842+0000 7fbaba612700 1 -- 172.21.15.98:0/1594998591 >> 172.21.15.98:0/1594998591 conn(0x7fbab40f86f0 msgr2=0x7fbab40fab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:03:03.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.842+0000 7fbaba612700 1 -- 172.21.15.98:0/1594998591 shutdown_connections 2024-04-22T21:03:03.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.842+0000 7fbaba612700 1 -- 172.21.15.98:0/1594998591 wait complete. 2024-04-22T21:03:03.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.842+0000 7fbaba612700 1 Processor -- start 2024-04-22T21:03:03.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.842+0000 7fbaba612700 1 -- start start 2024-04-22T21:03:03.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.843+0000 7fbaba612700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab4102ba0 0x7fbab41132a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:03:03.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.843+0000 7fbaba612700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab41137e0 con 0x7fbab4102ba0 2024-04-22T21:03:03.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.843+0000 7fbab3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab4102ba0 0x7fbab41132a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:03:03.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.843+0000 7fbab3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab4102ba0 0x7fbab41132a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.98:42502/0 (socket says 172.21.15.98:42502) 2024-04-22T21:03:03.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.843+0000 7fbab3fff700 1 -- 172.21.15.98:0/4217532848 learned_addr learned my addr 172.21.15.98:0/4217532848 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:03:03.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.843+0000 7fbab3fff700 1 -- 172.21.15.98:0/4217532848 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa4005040 con 0x7fbab4102ba0 2024-04-22T21:03:03.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.843+0000 7fbab3fff700 1 --2- 172.21.15.98:0/4217532848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab4102ba0 0x7fbab41132a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbaa400b820 tx=0x7fbaa4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:03:03.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.843+0000 7fbab0ff9700 1 -- 172.21.15.98:0/4217532848 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaa40093c0 con 0x7fbab4102ba0 2024-04-22T21:03:03.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.844+0000 7fbab0ff9700 1 -- 172.21.15.98:0/4217532848 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbaa4013420 con 0x7fbab4102ba0 2024-04-22T21:03:03.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.844+0000 7fbaba612700 1 -- 172.21.15.98:0/4217532848 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab41139e0 con 0x7fbab4102ba0 2024-04-22T21:03:03.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.844+0000 7fbab0ff9700 1 -- 172.21.15.98:0/4217532848 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaa401c5b0 con 0x7fbab4102ba0 2024-04-22T21:03:03.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.844+0000 7fbaba612700 1 -- 172.21.15.98:0/4217532848 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab4113ed0 con 0x7fbab4102ba0 2024-04-22T21:03:03.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.845+0000 7fbab0ff9700 1 -- 172.21.15.98:0/4217532848 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7fbaa4009520 con 0x7fbab4102ba0 2024-04-22T21:03:03.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.845+0000 7fbaba612700 1 -- 172.21.15.98:0/4217532848 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab40fe2f0 con 0x7fbab4102ba0 2024-04-22T21:03:03.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.845+0000 7fbab0ff9700 1 --2- 172.21.15.98:0/4217532848 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fba9c036970 0x7fba9c038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:03:03.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.845+0000 7fbab0ff9700 1 -- 172.21.15.98:0/4217532848 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fbaa4049aa0 con 0x7fbab4102ba0 2024-04-22T21:03:03.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.845+0000 7fbab37fe700 1 --2- 172.21.15.98:0/4217532848 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fba9c036970 0x7fba9c038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:03:03.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.846+0000 7fbab37fe700 1 --2- 172.21.15.98:0/4217532848 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fba9c036970 0x7fba9c038e20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8009990 tx=0x7fbaa8006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:03:03.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:03.850+0000 7fbab0ff9700 1 -- 172.21.15.98:0/4217532848 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fbaa4013a90 con 0x7fbab4102ba0 2024-04-22T21:03:04.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:03 smithi098 podman[33987]: 2024-04-22 21:03:03.984859347 +0000 UTC m=+0.594628104 container create e996b8c559ed943d855c0014ad237d358a1782438608d2773f6c9144756edd2d (image=quay.ceph.io/ceph-ci/ceph@sha256:3890cd726bd5e73ae623dabdf4ed76dd5df81c936b8bd8d761026babda969466, name=ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-mon-smithi098, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=quincy-70d0cb4, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-04-22T21:03:04.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:04.127+0000 7fbaba612700 1 -- 172.21.15.98:0/4217532848 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbab405f5f0 con 0x7fbab4102ba0 2024-04-22T21:03:04.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:04.128+0000 7fbab0ff9700 1 -- 172.21.15.98:0/4217532848 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fbaa4011370 con 0x7fbab4102ba0 2024-04-22T21:03:04.124 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:03:04.124 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:00:19.700503Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:03:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:04.131+0000 7fbaba612700 1 -- 172.21.15.98:0/4217532848 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fba9c036970 msgr2=0x7fba9c038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:04.131+0000 7fbaba612700 1 --2- 172.21.15.98:0/4217532848 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fba9c036970 0x7fba9c038e20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8009990 tx=0x7fbaa8006e40 comp rx=0 tx=0).stop 2024-04-22T21:03:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:04.131+0000 7fbaba612700 1 -- 172.21.15.98:0/4217532848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab4102ba0 msgr2=0x7fbab41132a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:04.131+0000 7fbaba612700 1 --2- 172.21.15.98:0/4217532848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab4102ba0 0x7fbab41132a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbaa400b820 tx=0x7fbaa4009e30 comp rx=0 tx=0).stop 2024-04-22T21:03:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:04.131+0000 7fbaba612700 1 -- 172.21.15.98:0/4217532848 shutdown_connections 2024-04-22T21:03:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:04.131+0000 7fbaba612700 1 --2- 172.21.15.98:0/4217532848 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fba9c036970 0x7fba9c038e20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:04.131+0000 7fbaba612700 1 --2- 172.21.15.98:0/4217532848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab4102ba0 0x7fbab41132a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:04.131+0000 7fbaba612700 1 -- 172.21.15.98:0/4217532848 >> 172.21.15.98:0/4217532848 conn(0x7fbab40f86f0 msgr2=0x7fbab40f9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:03:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:04.131+0000 7fbaba612700 1 -- 172.21.15.98:0/4217532848 shutdown_connections 2024-04-22T21:03:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:04.131+0000 7fbaba612700 1 -- 172.21.15.98:0/4217532848 wait complete. 2024-04-22T21:03:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-22T21:03:04.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 podman[33987]: 2024-04-22 21:03:04.287939861 +0000 UTC m=+0.897708609 container init e996b8c559ed943d855c0014ad237d358a1782438608d2773f6c9144756edd2d (image=quay.ceph.io/ceph-ci/ceph@sha256:3890cd726bd5e73ae623dabdf4ed76dd5df81c936b8bd8d761026babda969466, name=ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-mon-smithi098, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, RELEASE=quincy-70d0cb4, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-22T21:03:04.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 podman[33987]: 2024-04-22 21:03:04.291241103 +0000 UTC m=+0.901009858 container start e996b8c559ed943d855c0014ad237d358a1782438608d2773f6c9144756edd2d (image=quay.ceph.io/ceph-ci/ceph@sha256:3890cd726bd5e73ae623dabdf4ed76dd5df81c936b8bd8d761026babda969466, name=ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-mon-smithi098, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=quincy-70d0cb4, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-22T21:03:04.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T21:03:04.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: ceph version 17.2.7-884-g70d0cb4b (70d0cb4b9d576828afb0a05dcf8732894d343ece) quincy (stable), process ceph-mon, pid 2 2024-04-22T21:03:04.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pidfile_write: ignore empty --pid-file 2024-04-22T21:03:04.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: load: jerasure load: lrc 2024-04-22T21:03:04.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: RocksDB version: 6.15.5 2024-04-22T21:03:04.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-22T21:03:04.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Compile date Apr 22 2024 2024-04-22T21:03:04.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: DB SUMMARY 2024-04-22T21:03:04.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: DB Session ID: 8VZRWU3TQSGD88BUNF9A 2024-04-22T21:03:04.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: CURRENT file: CURRENT 2024-04-22T21:03:04.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T21:03:04.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-22T21:03:04.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi098/store.db dir, Total Num: 0, files: 2024-04-22T21:03:04.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi098/store.db: 000004.log size: 511 ; 2024-04-22T21:03:04.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.error_if_exists: 0 2024-04-22T21:03:04.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.create_if_missing: 0 2024-04-22T21:03:04.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.paranoid_checks: 1 2024-04-22T21:03:04.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T21:03:04.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.env: 0x564490eefb00 2024-04-22T21:03:04.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.fs: Posix File System 2024-04-22T21:03:04.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.info_log: 0x5644918e1c00 2024-04-22T21:03:04.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T21:03:04.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.statistics: (nil) 2024-04-22T21:03:04.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.use_fsync: 0 2024-04-22T21:03:04.754 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_log_file_size: 0 2024-04-22T21:03:04.754 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T21:03:04.754 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T21:03:04.755 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T21:03:04.755 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T21:03:04.755 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.allow_fallocate: 1 2024-04-22T21:03:04.755 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T21:03:04.755 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T21:03:04.755 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.use_direct_reads: 0 2024-04-22T21:03:04.755 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T21:03:04.755 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T21:03:04.755 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.db_log_dir: 2024-04-22T21:03:04.755 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi098/store.db 2024-04-22T21:03:04.755 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T21:03:04.755 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T21:03:04.755 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T21:03:04.756 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T21:03:04.756 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T21:03:04.756 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T21:03:04.756 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T21:03:04.756 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T21:03:04.756 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.write_buffer_manager: 0x564492656a20 2024-04-22T21:03:04.756 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T21:03:04.756 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-22T21:03:04.756 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T21:03:04.756 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T21:03:04.756 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T21:03:04.756 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T21:03:04.756 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T21:03:04.756 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T21:03:04.756 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T21:03:04.757 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.unordered_write: 0 2024-04-22T21:03:04.757 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T21:03:04.757 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T21:03:04.757 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T21:03:04.757 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T21:03:04.757 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.row_cache: None 2024-04-22T21:03:04.757 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.wal_filter: None 2024-04-22T21:03:04.757 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T21:03:04.757 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T21:03:04.757 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.preserve_deletes: 0 2024-04-22T21:03:04.757 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.two_write_queues: 0 2024-04-22T21:03:04.757 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T21:03:04.757 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.atomic_flush: 0 2024-04-22T21:03:04.757 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T21:03:04.758 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T21:03:04.758 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T21:03:04.758 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.log_readahead_size: 0 2024-04-22T21:03:04.758 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T21:03:04.758 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T21:03:04.758 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T21:03:04.758 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T21:03:04.758 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T21:03:04.758 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T21:03:04.758 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_background_jobs: 2 2024-04-22T21:03:04.758 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_background_compactions: -1 2024-04-22T21:03:04.758 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_subcompactions: 1 2024-04-22T21:03:04.758 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T21:03:04.758 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_open_files: -1 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_background_flushes: -1 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Compression algorithms supported: 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: kZSTD supported: 0 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: kXpressCompression supported: 0 2024-04-22T21:03:04.759 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T21:03:04.760 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: kZlibCompression supported: 1 2024-04-22T21:03:04.760 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: kSnappyCompression supported: 1 2024-04-22T21:03:04.760 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: kLZ4Compression supported: 1 2024-04-22T21:03:04.760 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: kBZip2Compression supported: 0 2024-04-22T21:03:04.760 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T21:03:04.760 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi098/store.db/MANIFEST-000003 2024-04-22T21:03:04.760 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-04-22T21:03:04.760 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T21:03:04.760 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.merge_operator: 2024-04-22T21:03:04.760 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compaction_filter: None 2024-04-22T21:03:04.760 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T21:03:04.760 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T21:03:04.760 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T21:03:04.760 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T21:03:04.760 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564491936f58) 2024-04-22T21:03:04.761 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-04-22T21:03:04.761 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T21:03:04.761 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T21:03:04.761 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-04-22T21:03:04.761 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: index_type: 0 2024-04-22T21:03:04.761 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: data_block_index_type: 0 2024-04-22T21:03:04.761 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: index_shortening: 1 2024-04-22T21:03:04.761 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T21:03:04.761 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: hash_index_allow_collision: 1 2024-04-22T21:03:04.761 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: checksum: 1 2024-04-22T21:03:04.761 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: no_block_cache: 0 2024-04-22T21:03:04.761 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: block_cache: 0x564491969090 2024-04-22T21:03:04.761 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-04-22T21:03:04.761 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: block_cache_options: 2024-04-22T21:03:04.761 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: capacity : 536870912 2024-04-22T21:03:04.761 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: num_shard_bits : 4 2024-04-22T21:03:04.762 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: strict_capacity_limit : 0 2024-04-22T21:03:04.762 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-04-22T21:03:04.762 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: block_cache_compressed: (nil) 2024-04-22T21:03:04.762 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: persistent_cache: (nil) 2024-04-22T21:03:04.762 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: block_size: 4096 2024-04-22T21:03:04.762 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: block_size_deviation: 10 2024-04-22T21:03:04.762 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: block_restart_interval: 16 2024-04-22T21:03:04.762 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: index_block_restart_interval: 1 2024-04-22T21:03:04.762 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: metadata_block_size: 4096 2024-04-22T21:03:04.762 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: partition_filters: 0 2024-04-22T21:03:04.762 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: use_delta_encoding: 1 2024-04-22T21:03:04.762 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-22T21:03:04.762 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: whole_key_filtering: 1 2024-04-22T21:03:04.762 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: verify_compression: 0 2024-04-22T21:03:04.763 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-04-22T21:03:04.763 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: format_version: 4 2024-04-22T21:03:04.763 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: enable_index_compression: 1 2024-04-22T21:03:04.763 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: block_align: 0 2024-04-22T21:03:04.763 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T21:03:04.763 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T21:03:04.763 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compression: NoCompression 2024-04-22T21:03:04.763 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T21:03:04.763 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T21:03:04.763 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T21:03:04.763 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.num_levels: 7 2024-04-22T21:03:04.763 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T21:03:04.763 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T21:03:04.763 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T21:03:04.763 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T21:03:04.763 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T21:03:04.764 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T21:03:04.765 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T21:03:04.765 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T21:03:04.765 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T21:03:04.765 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T21:03:04.765 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T21:03:04.765 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T21:03:04.765 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T21:03:04.765 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T21:03:04.765 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T21:03:04.765 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T21:03:04.765 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T21:03:04.765 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T21:03:04.765 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.arena_block_size: 4194304 2024-04-22T21:03:04.765 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T21:03:04.766 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T21:03:04.766 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-22T21:03:04.766 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T21:03:04.766 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T21:03:04.766 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T21:03:04.766 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T21:03:04.766 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T21:03:04.766 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T21:03:04.766 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T21:03:04.766 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T21:03:04.766 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T21:03:04.766 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T21:03:04.766 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T21:03:04.766 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.table_properties_collectors: 2024-04-22T21:03:04.767 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.inplace_update_support: 0 2024-04-22T21:03:04.767 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T21:03:04.767 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T21:03:04.767 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T21:03:04.767 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T21:03:04.767 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.bloom_locality: 0 2024-04-22T21:03:04.767 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.max_successive_merges: 0 2024-04-22T21:03:04.767 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T21:03:04.767 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T21:03:04.767 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T21:03:04.767 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T21:03:04.767 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.ttl: 2592000 2024-04-22T21:03:04.767 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T21:03:04.767 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.enable_blob_files: false 2024-04-22T21:03:04.767 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.min_blob_size: 0 2024-04-22T21:03:04.768 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T21:03:04.768 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T21:03:04.768 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T21:03:04.768 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T21:03:04.768 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi098/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T21:03:04.768 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-04-22T21:03:04.768 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-04-22T21:03:04.768 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713819784317138, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-22T21:03:04.768 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-04-22T21:03:04.768 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-22T21:03:04.768 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713819784317447, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713819784, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "f2a78417-564a-44ae-9a44-95937dbc71dd", "db_session_id": "8VZRWU3TQSGD88BUNF9A"}} 2024-04-22T21:03:04.768 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-04-22T21:03:04.768 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713819784317792, "job": 1, "event": "recovery_finished"} 2024-04-22T21:03:04.768 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi098/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T21:03:04.768 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5644918fd180 2024-04-22T21:03:04.769 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: DB pointer 0x5644919d2000 2024-04-22T21:03:04.769 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-04-22T21:03:04.769 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-22T21:03:04.769 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: ** DB Stats ** 2024-04-22T21:03:04.769 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:03:04.769 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T21:03:04.769 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T21:03:04.769 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T21:03:04.769 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T21:03:04.769 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-22T21:03:04.769 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T21:03:04.769 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: 2024-04-22T21:03:04.769 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: ** Compaction Stats [default] ** 2024-04-22T21:03:04.769 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:03:04.769 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:03:04.769 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-22T21:03:04.770 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-22T21:03:04.770 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-22T21:03:04.770 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: 2024-04-22T21:03:04.770 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: ** Compaction Stats [default] ** 2024-04-22T21:03:04.770 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:03:04.770 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:03:04.770 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-22T21:03:04.770 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:03:04.770 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T21:03:04.770 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T21:03:04.770 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T21:03:04.770 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T21:03:04.770 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T21:03:04.770 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Cumulative compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:03:04.771 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Interval compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:03:04.771 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T21:03:04.771 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: 2024-04-22T21:03:04.771 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T21:03:04.771 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: 2024-04-22T21:03:04.771 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: ** Compaction Stats [default] ** 2024-04-22T21:03:04.771 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:03:04.771 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:03:04.771 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-22T21:03:04.771 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-22T21:03:04.771 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-22T21:03:04.771 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: 2024-04-22T21:03:04.771 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: ** Compaction Stats [default] ** 2024-04-22T21:03:04.771 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:03:04.772 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:03:04.772 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-22T21:03:04.772 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:03:04.772 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T21:03:04.772 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T21:03:04.772 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T21:03:04.772 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T21:03:04.772 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T21:03:04.772 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Cumulative compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:03:04.772 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:03:04.772 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T21:03:04.772 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: 2024-04-22T21:03:04.772 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T21:03:04.772 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mon.smithi098 does not exist in monmap, will attempt to join an existing cluster 2024-04-22T21:03:04.772 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: using public_addr v2:172.21.15.98:0/0 -> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-04-22T21:03:04.772 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: starting mon.smithi098 rank -1 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi098 fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T21:03:04.773 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mon.smithi098@-1(???) e0 preinit fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T21:03:04.773 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mon.smithi098@-1(synchronizing).mds e1 new map 2024-04-22T21:03:04.773 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mon.smithi098@-1(synchronizing).mds e1 print_map 2024-04-22T21:03:04.773 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: e1 2024-04-22T21:03:04.773 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T21:03:04.773 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T21:03:04.773 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: legacy client fscid: -1 2024-04-22T21:03:04.773 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: 2024-04-22T21:03:04.773 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout: No filesystems configured 2024-04-22T21:03:04.773 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mon.smithi098@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-22T21:03:04.773 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mon.smithi098@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-22T21:03:04.773 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mon.smithi098@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-22T21:03:04.773 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mon.smithi098@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-22T21:03:04.774 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mon.smithi098@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-22T21:03:04.774 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mon.smithi098@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-22T21:03:04.774 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mon.smithi098@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-04-22T21:03:04.774 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mon.smithi098@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T21:03:04.774 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mon.smithi098@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:03:04.774 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mon.smithi098@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:03:04.774 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mon.smithi098@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:03:04.774 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Deploying daemon crash.smithi038 on smithi038 2024-04-22T21:03:04.774 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:03:04.774 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.774 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.774 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.774 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.774 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.775 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Deploying daemon node-exporter.smithi038 on smithi038 2024-04-22T21:03:04.775 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:03:04.775 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.775 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.775 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Deploying cephadm binary to smithi098 2024-04-22T21:03:04.775 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.775 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Added host smithi098 2024-04-22T21:03:04.775 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.775 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.775 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.775 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.775 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.775 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Deploying daemon alertmanager.smithi038 on smithi038 2024-04-22T21:03:04.775 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.775 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:03:04.776 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.776 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.776 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.776 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/3416013377' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T21:03:04.776 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/3416013377' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T21:03:04.776 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T21:03:04.776 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.776 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.776 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.776 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi038:172.21.15.38=smithi038;smithi098:172.21.15.98=smithi098", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:03:04.776 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Saving service mon spec with placement smithi038:172.21.15.38=smithi038;smithi098:172.21.15.98=smithi098;count:2 2024-04-22T21:03:04.776 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.776 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.776 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.776 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.776 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.777 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.777 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T21:03:04.777 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T21:03:04.777 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.777 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Deploying daemon grafana.smithi038 on smithi038 2024-04-22T21:03:04.777 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.777 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/1600156776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.777 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.777 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.777 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.777 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/3343806550' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.777 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.777 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/2123601850' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.777 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.777 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.777 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/454117421' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.778 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.778 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.778 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.778 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.778 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.778 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.778 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.778 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.778 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Deploying daemon prometheus.smithi038 on smithi038 2024-04-22T21:03:04.778 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/3153398667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.778 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.778 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.778 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.778 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/2814541345' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.779 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.779 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.779 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/158392414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.779 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.779 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.779 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.779 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.779 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T21:03:04.779 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/2375417514' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.779 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.779 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14162 172.21.15.38:0/2661966969' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T21:03:04.779 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mgrmap e14: smithi038.lxfgut(active, since 60s) 2024-04-22T21:03:04.779 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/2749463796' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.779 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/605793336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.779 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Active manager daemon smithi038.lxfgut restarted 2024-04-22T21:03:04.780 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Activating manager daemon smithi038.lxfgut 2024-04-22T21:03:04.780 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T21:03:04.780 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mgrmap e15: smithi038.lxfgut(active, starting, since 0.00475133s) 2024-04-22T21:03:04.780 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2024-04-22T21:03:04.780 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr metadata", "who": "smithi038.lxfgut", "id": "smithi038.lxfgut"}]: dispatch 2024-04-22T21:03:04.780 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T21:03:04.780 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T21:03:04.780 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T21:03:04.780 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Manager daemon smithi038.lxfgut is now available 2024-04-22T21:03:04.780 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.780 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.780 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.780 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:03:04.780 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.lxfgut/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:03:04.780 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.lxfgut/trash_purge_schedule"}]: dispatch 2024-04-22T21:03:04.781 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.781 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mgrmap e16: smithi038.lxfgut(active, since 1.00844s) 2024-04-22T21:03:04.781 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.781 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: [22/Apr/2024:21:02:39] ENGINE Bus STARTING 2024-04-22T21:03:04.781 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: [22/Apr/2024:21:02:39] ENGINE Serving on https://172.21.15.38:7150 2024-04-22T21:03:04.781 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: [22/Apr/2024:21:02:39] ENGINE Bus STARTED 2024-04-22T21:03:04.781 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.781 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.781 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mgrmap e17: smithi038.lxfgut(active, since 2s) 2024-04-22T21:03:04.781 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/2295705493' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.781 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.781 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.781 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.781 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.782 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:03:04.782 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.782 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.782 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.782 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.782 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:03:04.782 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:04.782 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:03:04.782 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-22T21:03:04.782 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Updating smithi098:/etc/ceph/ceph.conf 2024-04-22T21:03:04.782 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Updating smithi098:/var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T21:03:04.782 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Updating smithi038:/var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T21:03:04.782 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:03:04.782 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:03:04.783 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Updating smithi098:/var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T21:03:04.783 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Updating smithi038:/var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T21:03:04.783 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.783 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.783 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.783 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.783 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.783 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi098", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T21:03:04.783 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi098", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-22T21:03:04.783 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:04.783 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Deploying daemon crash.smithi098 on smithi098 2024-04-22T21:03:04.783 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/202986320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.783 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/2244039159' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.783 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.783 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.784 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.784 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.784 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Deploying daemon node-exporter.smithi098 on smithi098 2024-04-22T21:03:04.784 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.784 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:03:04.784 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/439004432' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.784 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.784 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.784 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.784 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.784 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi098.azqohf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:03:04.785 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi098.azqohf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T21:03:04.785 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:03:04.785 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:04.785 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Deploying daemon mgr.smithi098.azqohf on smithi098 2024-04-22T21:03:04.785 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.785 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.785 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.785 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.785 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.785 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.785 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:03:04.785 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:04.785 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: Deploying daemon mon.smithi098 on smithi098 2024-04-22T21:03:04.785 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.786 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/2892043602' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:04.786 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:04.786 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:04.786 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 ceph-mon[34053]: mon.smithi098@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-04-22T21:03:04.786 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 bash[33987]: e996b8c559ed943d855c0014ad237d358a1782438608d2773f6c9144756edd2d 2024-04-22T21:03:04.786 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:04 smithi098 systemd[1]: Started Ceph mon.smithi098 for 3c5f5352-00eb-11ef-bc93-c7b262605968. 2024-04-22T21:03:06.072 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:03:06.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:03:06.322 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi098/config 2024-04-22T21:03:09.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:09.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2024-04-22T21:03:09.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: mon.smithi038 calling monitor election 2024-04-22T21:03:09.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:03:09.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: from='mgr.? 172.21.15.98:0/1913867439' entity='mgr.smithi098.azqohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi098.azqohf/crt"}]: dispatch 2024-04-22T21:03:09.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:03:09.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:09.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:03:09.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: mon.smithi098 calling monitor election 2024-04-22T21:03:09.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:03:09.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:09.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:03:09.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:03:09.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:03:09.751 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: mon.smithi038 is new leader, mons smithi038,smithi098 in quorum (ranks 0,1) 2024-04-22T21:03:09.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: monmap e2: 2 mons at {smithi038=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],smithi098=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-22T21:03:09.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: fsmap 2024-04-22T21:03:09.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T21:03:09.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: mgrmap e17: smithi038.lxfgut(active, since 31s) 2024-04-22T21:03:09.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: overall HEALTH_OK 2024-04-22T21:03:09.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: Standby manager daemon smithi098.azqohf started 2024-04-22T21:03:09.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: from='mgr.? 172.21.15.98:0/1913867439' entity='mgr.smithi098.azqohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T21:03:09.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:09.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: from='mgr.? 172.21.15.98:0/1913867439' entity='mgr.smithi098.azqohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi098.azqohf/key"}]: dispatch 2024-04-22T21:03:09.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: from='mgr.? 172.21.15.98:0/1913867439' entity='mgr.smithi098.azqohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T21:03:09.752 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:09 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:09.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:09.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2024-04-22T21:03:09.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: mon.smithi038 calling monitor election 2024-04-22T21:03:09.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:03:09.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: from='mgr.? 172.21.15.98:0/1913867439' entity='mgr.smithi098.azqohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi098.azqohf/crt"}]: dispatch 2024-04-22T21:03:09.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:03:09.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:09.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:03:09.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: mon.smithi098 calling monitor election 2024-04-22T21:03:09.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:03:09.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:09.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:03:09.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:03:09.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:03:09.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: mon.smithi038 is new leader, mons smithi038,smithi098 in quorum (ranks 0,1) 2024-04-22T21:03:09.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: monmap e2: 2 mons at {smithi038=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],smithi098=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-22T21:03:09.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: fsmap 2024-04-22T21:03:09.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T21:03:09.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: mgrmap e17: smithi038.lxfgut(active, since 31s) 2024-04-22T21:03:09.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: overall HEALTH_OK 2024-04-22T21:03:09.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: Standby manager daemon smithi098.azqohf started 2024-04-22T21:03:09.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: from='mgr.? 172.21.15.98:0/1913867439' entity='mgr.smithi098.azqohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T21:03:09.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:09.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: from='mgr.? 172.21.15.98:0/1913867439' entity='mgr.smithi098.azqohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi098.azqohf/key"}]: dispatch 2024-04-22T21:03:09.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: from='mgr.? 172.21.15.98:0/1913867439' entity='mgr.smithi098.azqohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T21:03:09.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:09 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:10.691 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:10 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:10.691 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:10 smithi098 ceph-mon[34053]: mgrmap e18: smithi038.lxfgut(active, since 31s), standbys: smithi098.azqohf 2024-04-22T21:03:10.691 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:10 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr metadata", "who": "smithi098.azqohf", "id": "smithi098.azqohf"}]: dispatch 2024-04-22T21:03:10.691 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:10 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:10.691 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:10 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:10.692 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:10 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:03:10.692 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:10 smithi098 ceph-mon[34053]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:10.692 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:10 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:03:10.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.725+0000 7fe318539700 1 -- 172.21.15.98:0/355951381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc003580 msgr2=0x7fe2fc005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:10.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.725+0000 7fe318539700 1 --2- 172.21.15.98:0/355951381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc003580 0x7fe2fc005a00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe3100fcb00 tx=0x7fe30c00b040 comp rx=0 tx=0).stop 2024-04-22T21:03:10.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.725+0000 7fe318539700 1 -- 172.21.15.98:0/355951381 shutdown_connections 2024-04-22T21:03:10.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.725+0000 7fe318539700 1 --2- 172.21.15.98:0/355951381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc003580 0x7fe2fc005a00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.725+0000 7fe318539700 1 -- 172.21.15.98:0/355951381 >> 172.21.15.98:0/355951381 conn(0x7fe3100f8730 msgr2=0x7fe3100fab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:03:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.725+0000 7fe318539700 1 -- 172.21.15.98:0/355951381 shutdown_connections 2024-04-22T21:03:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.725+0000 7fe318539700 1 -- 172.21.15.98:0/355951381 wait complete. 2024-04-22T21:03:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.726+0000 7fe318539700 1 Processor -- start 2024-04-22T21:03:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.726+0000 7fe318539700 1 -- start start 2024-04-22T21:03:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.726+0000 7fe318539700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc003580 0x7fe3101132b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:03:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.726+0000 7fe318539700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3101137f0 0x7fe31010d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:03:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.726+0000 7fe318539700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe310113e00 con 0x7fe2fc003580 2024-04-22T21:03:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.726+0000 7fe318539700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31010d750 con 0x7fe3101137f0 2024-04-22T21:03:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.727+0000 7fe315ad4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3101137f0 0x7fe31010d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:03:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.727+0000 7fe315ad4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3101137f0 0x7fe31010d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55516/0 (socket says 172.21.15.98:55516) 2024-04-22T21:03:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.727+0000 7fe315ad4700 1 -- 172.21.15.98:0/270205265 learned_addr learned my addr 172.21.15.98:0/270205265 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:03:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.727+0000 7fe315ad4700 1 -- 172.21.15.98:0/270205265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3101137f0 msgr2=0x7fe31010d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T21:03:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.727+0000 7fe315ad4700 1 -- 172.21.15.98:0/270205265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3101137f0 msgr2=0x7fe31010d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T21:03:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.727+0000 7fe315ad4700 1 --2- 172.21.15.98:0/270205265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3101137f0 0x7fe31010d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-22T21:03:10.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.727+0000 7fe315ad4700 1 --2- 172.21.15.98:0/270205265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3101137f0 0x7fe31010d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T21:03:10.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.727+0000 7fe3162d5700 1 --2- 172.21.15.98:0/270205265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc003580 0x7fe3101132b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:03:10.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.727+0000 7fe3162d5700 1 -- 172.21.15.98:0/270205265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3101137f0 msgr2=0x7fe31010d210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T21:03:10.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.727+0000 7fe3162d5700 1 --2- 172.21.15.98:0/270205265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3101137f0 0x7fe31010d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:10.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.727+0000 7fe3162d5700 1 -- 172.21.15.98:0/270205265 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c005040 con 0x7fe2fc003580 2024-04-22T21:03:10.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.727+0000 7fe3162d5700 1 --2- 172.21.15.98:0/270205265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc003580 0x7fe3101132b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe30c000c00 tx=0x7fe30c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:03:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.728+0000 7fe306ffd700 1 -- 172.21.15.98:0/270205265 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe30c01b070 con 0x7fe2fc003580 2024-04-22T21:03:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.728+0000 7fe318539700 1 -- 172.21.15.98:0/270205265 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31010d9d0 con 0x7fe2fc003580 2024-04-22T21:03:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.728+0000 7fe306ffd700 1 -- 172.21.15.98:0/270205265 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe30c016cf0 con 0x7fe2fc003580 2024-04-22T21:03:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.728+0000 7fe306ffd700 1 -- 172.21.15.98:0/270205265 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe30c01f6b0 con 0x7fe2fc003580 2024-04-22T21:03:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.728+0000 7fe318539700 1 -- 172.21.15.98:0/270205265 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31010df10 con 0x7fe2fc003580 2024-04-22T21:03:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.729+0000 7fe318539700 1 -- 172.21.15.98:0/270205265 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe310101680 con 0x7fe2fc003580 2024-04-22T21:03:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.730+0000 7fe306ffd700 1 -- 172.21.15.98:0/270205265 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7fe30c020aa0 con 0x7fe2fc003580 2024-04-22T21:03:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.730+0000 7fe306ffd700 1 --2- 172.21.15.98:0/270205265 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe2f4069320 0x7fe2f406b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:03:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.730+0000 7fe306ffd700 1 -- 172.21.15.98:0/270205265 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fe30c087e30 con 0x7fe2fc003580 2024-04-22T21:03:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.730+0000 7fe315ad4700 1 --2- 172.21.15.98:0/270205265 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe2f4069320 0x7fe2f406b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:03:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.731+0000 7fe315ad4700 1 --2- 172.21.15.98:0/270205265 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe2f4069320 0x7fe2f406b7d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe300006fd0 tx=0x7fe300008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:03:10.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:10.735+0000 7fe306ffd700 1 -- 172.21.15.98:0/270205265 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe30c058c50 con 0x7fe2fc003580 2024-04-22T21:03:10.776 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:10 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:10.776 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:10 smithi038 ceph-mon[26374]: mgrmap e18: smithi038.lxfgut(active, since 31s), standbys: smithi098.azqohf 2024-04-22T21:03:10.776 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:10 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr metadata", "who": "smithi098.azqohf", "id": "smithi098.azqohf"}]: dispatch 2024-04-22T21:03:10.776 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:10 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:10.776 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:10 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:10.776 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:10 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:03:10.777 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:10 smithi038 ceph-mon[26374]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:10.777 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:10 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:03:11.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:11.012+0000 7fe318539700 1 -- 172.21.15.98:0/270205265 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3101048e0 con 0x7fe2fc003580 2024-04-22T21:03:11.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:11.013+0000 7fe306ffd700 1 -- 172.21.15.98:0/270205265 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1013 (secure 0 0 0) 0x7fe30c024090 con 0x7fe2fc003580 2024-04-22T21:03:11.009 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-22T21:03:11.009 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":2,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","modified":"2024-04-22T21:03:04.344847Z","created":"2024-04-22T21:00:19.700503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi098","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T21:03:11.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:11.015+0000 7fe318539700 1 -- 172.21.15.98:0/270205265 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe2f4069320 msgr2=0x7fe2f406b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:11.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:11.015+0000 7fe318539700 1 --2- 172.21.15.98:0/270205265 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe2f4069320 0x7fe2f406b7d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe300006fd0 tx=0x7fe300008040 comp rx=0 tx=0).stop 2024-04-22T21:03:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:11.015+0000 7fe318539700 1 -- 172.21.15.98:0/270205265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc003580 msgr2=0x7fe3101132b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:11.015+0000 7fe318539700 1 --2- 172.21.15.98:0/270205265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc003580 0x7fe3101132b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe30c000c00 tx=0x7fe30c00b040 comp rx=0 tx=0).stop 2024-04-22T21:03:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:11.015+0000 7fe318539700 1 -- 172.21.15.98:0/270205265 shutdown_connections 2024-04-22T21:03:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:11.015+0000 7fe318539700 1 --2- 172.21.15.98:0/270205265 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe2f4069320 0x7fe2f406b7d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:11.015+0000 7fe318539700 1 --2- 172.21.15.98:0/270205265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc003580 0x7fe3101132b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:11.015+0000 7fe318539700 1 --2- 172.21.15.98:0/270205265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3101137f0 0x7fe31010d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:11.015+0000 7fe318539700 1 -- 172.21.15.98:0/270205265 >> 172.21.15.98:0/270205265 conn(0x7fe3100f8730 msgr2=0x7fe3100fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:03:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:11.015+0000 7fe318539700 1 -- 172.21.15.98:0/270205265 shutdown_connections 2024-04-22T21:03:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:03:11.015+0000 7fe318539700 1 -- 172.21.15.98:0/270205265 wait complete. 2024-04-22T21:03:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 2 2024-04-22T21:03:11.583 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:11 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/270205265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:11.834 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-22T21:03:11.834 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-22T21:03:11.857 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:11 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/270205265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:03:12.108 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:03:13.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:13.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:13.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:13.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:03:13.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-22T21:03:13.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: Updating smithi098:/etc/ceph/ceph.conf 2024-04-22T21:03:13.251 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:13.251 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: Updating smithi038:/var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T21:03:13.251 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: Updating smithi098:/var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T21:03:13.251 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:13.251 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:13.251 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:13.251 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:13.251 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:13.251 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:03:13.251 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T21:03:13.251 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:12 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:13.266 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:13.266 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:13.266 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:13.266 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:03:13.266 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-22T21:03:13.266 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: Updating smithi098:/etc/ceph/ceph.conf 2024-04-22T21:03:13.266 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:13.266 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: Updating smithi038:/var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T21:03:13.266 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: Updating smithi098:/var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T21:03:13.266 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:13.266 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:13.266 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:13.267 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:13.267 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:13.267 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:03:13.267 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T21:03:13.267 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:12 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:14.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:14 smithi038 ceph-mon[26374]: Reconfiguring mon.smithi038 (unknown last config time)... 2024-04-22T21:03:14.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:14 smithi038 ceph-mon[26374]: Reconfiguring daemon mon.smithi038 on smithi038 2024-04-22T21:03:14.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:14 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:14.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:14 smithi098 ceph-mon[34053]: Reconfiguring mon.smithi038 (unknown last config time)... 2024-04-22T21:03:14.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:14 smithi098 ceph-mon[34053]: Reconfiguring daemon mon.smithi038 on smithi038 2024-04-22T21:03:14.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:14 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:14.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.827+0000 7fa9effad700 1 -- 172.21.15.38:0/1489009576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9e8102e60 msgr2=0x7fa9e8103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:14.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.827+0000 7fa9effad700 1 --2- 172.21.15.38:0/1489009576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9e8102e60 0x7fa9e8103230 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009ce0 tx=0x7fa9d8004d10 comp rx=0 tx=0).stop 2024-04-22T21:03:14.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.827+0000 7fa9effad700 1 -- 172.21.15.38:0/1489009576 shutdown_connections 2024-04-22T21:03:14.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.827+0000 7fa9effad700 1 --2- 172.21.15.38:0/1489009576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e80656b0 0x7fa9e8065b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:14.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.827+0000 7fa9effad700 1 --2- 172.21.15.38:0/1489009576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9e8102e60 0x7fa9e8103230 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:14.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.827+0000 7fa9effad700 1 -- 172.21.15.38:0/1489009576 >> 172.21.15.38:0/1489009576 conn(0x7fa9e8072bc0 msgr2=0x7fa9e8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:03:14.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.827+0000 7fa9effad700 1 -- 172.21.15.38:0/1489009576 shutdown_connections 2024-04-22T21:03:14.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.827+0000 7fa9effad700 1 -- 172.21.15.38:0/1489009576 wait complete. 2024-04-22T21:03:14.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.828+0000 7fa9effad700 1 Processor -- start 2024-04-22T21:03:14.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.828+0000 7fa9effad700 1 -- start start 2024-04-22T21:03:14.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.828+0000 7fa9effad700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9e80656b0 0x7fa9e8113d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:03:14.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.828+0000 7fa9effad700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e8102e60 0x7fa9e8114290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:03:14.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.828+0000 7fa9effad700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e81148e0 con 0x7fa9e80656b0 2024-04-22T21:03:14.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.828+0000 7fa9effad700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e810dcb0 con 0x7fa9e8102e60 2024-04-22T21:03:14.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.828+0000 7fa9edd49700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9e80656b0 0x7fa9e8113d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:03:14.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.828+0000 7fa9edd49700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9e80656b0 0x7fa9e8113d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51984/0 (socket says 172.21.15.38:51984) 2024-04-22T21:03:14.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.828+0000 7fa9edd49700 1 -- 172.21.15.38:0/1865984185 learned_addr learned my addr 172.21.15.38:0/1865984185 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:03:14.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.828+0000 7fa9ed548700 1 --2- 172.21.15.38:0/1865984185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e8102e60 0x7fa9e8114290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:03:14.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.828+0000 7fa9edd49700 1 -- 172.21.15.38:0/1865984185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e8102e60 msgr2=0x7fa9e8114290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:14.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.828+0000 7fa9edd49700 1 --2- 172.21.15.38:0/1865984185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e8102e60 0x7fa9e8114290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:14.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.828+0000 7fa9edd49700 1 -- 172.21.15.38:0/1865984185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d8009940 con 0x7fa9e80656b0 2024-04-22T21:03:14.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.828+0000 7fa9edd49700 1 --2- 172.21.15.38:0/1865984185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9e80656b0 0x7fa9e8113d50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8005010 tx=0x7fa9d8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:03:14.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.829+0000 7fa9e67fc700 1 -- 172.21.15.38:0/1865984185 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9d800e040 con 0x7fa9e80656b0 2024-04-22T21:03:14.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.829+0000 7fa9e67fc700 1 -- 172.21.15.38:0/1865984185 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9d8008e20 con 0x7fa9e80656b0 2024-04-22T21:03:14.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.829+0000 7fa9effad700 1 -- 172.21.15.38:0/1865984185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e810df30 con 0x7fa9e80656b0 2024-04-22T21:03:14.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.829+0000 7fa9e67fc700 1 -- 172.21.15.38:0/1865984185 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9d801c660 con 0x7fa9e80656b0 2024-04-22T21:03:14.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.829+0000 7fa9effad700 1 -- 172.21.15.38:0/1865984185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e810e4f0 con 0x7fa9e80656b0 2024-04-22T21:03:14.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.830+0000 7fa9effad700 1 -- 172.21.15.38:0/1865984185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e8100130 con 0x7fa9e80656b0 2024-04-22T21:03:14.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.830+0000 7fa9e67fc700 1 -- 172.21.15.38:0/1865984185 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7fa9d8012a70 con 0x7fa9e80656b0 2024-04-22T21:03:14.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.830+0000 7fa9e67fc700 1 --2- 172.21.15.38:0/1865984185 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa9d0069210 0x7fa9d006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:03:14.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.831+0000 7fa9e67fc700 1 -- 172.21.15.38:0/1865984185 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fa9d8087c20 con 0x7fa9e80656b0 2024-04-22T21:03:14.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.831+0000 7fa9ed548700 1 --2- 172.21.15.38:0/1865984185 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa9d0069210 0x7fa9d006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:03:14.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.831+0000 7fa9ed548700 1 --2- 172.21.15.38:0/1865984185 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa9d0069210 0x7fa9d006b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc007fd0 tx=0x7fa9dc009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:03:14.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:14.835+0000 7fa9e67fc700 1 -- 172.21.15.38:0/1865984185 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa9d8058ac0 con 0x7fa9e80656b0 2024-04-22T21:03:15.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:15.033+0000 7fa9effad700 1 -- 172.21.15.38:0/1865984185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa9e8109ee0 con 0x7fa9e80656b0 2024-04-22T21:03:15.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:15.033+0000 7fa9e67fc700 1 -- 172.21.15.38:0/1865984185 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+223 (secure 0 0 0) 0x7fa9d80873d0 con 0x7fa9e80656b0 2024-04-22T21:03:15.029 INFO:teuthology.orchestra.run.smithi038.stdout:# minimal ceph.conf for 3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T21:03:15.029 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2024-04-22T21:03:15.029 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = 3c5f5352-00eb-11ef-bc93-c7b262605968 2024-04-22T21:03:15.029 INFO:teuthology.orchestra.run.smithi038.stdout: mon_host = [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-04-22T21:03:15.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:15.035+0000 7fa9effad700 1 -- 172.21.15.38:0/1865984185 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa9d0069210 msgr2=0x7fa9d006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:15.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:15.035+0000 7fa9effad700 1 --2- 172.21.15.38:0/1865984185 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa9d0069210 0x7fa9d006b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc007fd0 tx=0x7fa9dc009040 comp rx=0 tx=0).stop 2024-04-22T21:03:15.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:15.035+0000 7fa9effad700 1 -- 172.21.15.38:0/1865984185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9e80656b0 msgr2=0x7fa9e8113d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:15.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:15.035+0000 7fa9effad700 1 --2- 172.21.15.38:0/1865984185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9e80656b0 0x7fa9e8113d50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8005010 tx=0x7fa9d8004d10 comp rx=0 tx=0).stop 2024-04-22T21:03:15.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:15.035+0000 7fa9effad700 1 -- 172.21.15.38:0/1865984185 shutdown_connections 2024-04-22T21:03:15.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:15.035+0000 7fa9effad700 1 --2- 172.21.15.38:0/1865984185 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa9d0069210 0x7fa9d006b6c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:15.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:15.035+0000 7fa9effad700 1 --2- 172.21.15.38:0/1865984185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9e80656b0 0x7fa9e8113d50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:15.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:15.035+0000 7fa9effad700 1 --2- 172.21.15.38:0/1865984185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e8102e60 0x7fa9e8114290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:15.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:15.035+0000 7fa9effad700 1 -- 172.21.15.38:0/1865984185 >> 172.21.15.38:0/1865984185 conn(0x7fa9e8072bc0 msgr2=0x7fa9e80fb160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:03:15.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:15.035+0000 7fa9effad700 1 -- 172.21.15.38:0/1865984185 shutdown_connections 2024-04-22T21:03:15.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:15.035+0000 7fa9effad700 1 -- 172.21.15.38:0/1865984185 wait complete. 2024-04-22T21:03:15.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:15 smithi038 ceph-mon[26374]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:15.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:15 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:15.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:15 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:15.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:15 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi038.lxfgut", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:03:15.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:15 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:03:15.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:15 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:15.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:15 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1865984185' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:15.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:15 smithi098 ceph-mon[34053]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:15.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:15 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:15.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:15 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:15.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:15 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi038.lxfgut", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:03:15.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:15 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:03:15.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:15 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:15.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:15 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/1865984185' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:15.943 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-22T21:03:15.943 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T21:03:15.943 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T21:03:16.012 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T21:03:16.012 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:03:16.078 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T21:03:16.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T21:03:16.111 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T21:03:16.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:03:16.177 INFO:tasks.cephadm:Deploying OSDs... 2024-04-22T21:03:16.177 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T21:03:16.178 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T21:03:16.194 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T21:03:16.195 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2024-04-22T21:03:16.251 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T21:03:16.251 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:03:16.251 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-22T21:03:16.251 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:03:16.252 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:03:16.252 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-22 21:02:29.620868659 +0000 2024-04-22T21:03:16.252 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-22 20:56:58.892173690 +0000 2024-04-22T21:03:16.252 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-22 20:56:58.892173690 +0000 2024-04-22T21:03:16.252 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-22 20:56:58.892173690 +0000 2024-04-22T21:03:16.252 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T21:03:16.315 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-22T21:03:16.316 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-22T21:03:16.316 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000289305 s, 1.8 MB/s 2024-04-22T21:03:16.317 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T21:03:16.374 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2024-04-22T21:03:16.431 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T21:03:16.431 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:03:16.431 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-22T21:03:16.431 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:03:16.431 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:03:16.431 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-22 21:02:29.620868659 +0000 2024-04-22T21:03:16.431 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-22 20:56:59.254166950 +0000 2024-04-22T21:03:16.431 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-22 20:56:59.254166950 +0000 2024-04-22T21:03:16.431 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-22 20:56:59.254166950 +0000 2024-04-22T21:03:16.432 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T21:03:16.495 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-22T21:03:16.495 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-22T21:03:16.495 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000277452 s, 1.8 MB/s 2024-04-22T21:03:16.496 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T21:03:16.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:16 smithi098 ceph-mon[34053]: Reconfiguring mgr.smithi038.lxfgut (unknown last config time)... 2024-04-22T21:03:16.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:16 smithi098 ceph-mon[34053]: Reconfiguring daemon mgr.smithi038.lxfgut on smithi038 2024-04-22T21:03:16.553 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2024-04-22T21:03:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T21:03:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:03:16.609 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-22T21:03:16.609 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:03:16.609 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:03:16.610 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-22 21:02:29.621868641 +0000 2024-04-22T21:03:16.610 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-22 20:56:59.616160208 +0000 2024-04-22T21:03:16.610 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-22 20:56:59.616160208 +0000 2024-04-22T21:03:16.610 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-22 20:56:59.616160208 +0000 2024-04-22T21:03:16.610 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T21:03:16.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:16 smithi038 ceph-mon[26374]: Reconfiguring mgr.smithi038.lxfgut (unknown last config time)... 2024-04-22T21:03:16.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:16 smithi038 ceph-mon[26374]: Reconfiguring daemon mgr.smithi038.lxfgut on smithi038 2024-04-22T21:03:16.647 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-22T21:03:16.647 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-22T21:03:16.648 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.00032429 s, 1.6 MB/s 2024-04-22T21:03:16.649 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T21:03:16.705 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2024-04-22T21:03:16.761 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T21:03:16.762 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:03:16.762 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-22T21:03:16.762 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:03:16.762 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:03:16.762 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-22 21:02:29.621868641 +0000 2024-04-22T21:03:16.762 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-22 20:56:59.975153523 +0000 2024-04-22T21:03:16.762 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-22 20:56:59.975153523 +0000 2024-04-22T21:03:16.762 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-22 20:56:59.975153523 +0000 2024-04-22T21:03:16.762 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T21:03:16.825 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-22T21:03:16.825 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-22T21:03:16.825 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000258971 s, 2.0 MB/s 2024-04-22T21:03:16.826 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T21:03:16.882 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T21:03:16.883 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T21:03:16.901 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T21:03:16.902 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-04-22T21:03:16.959 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T21:03:16.959 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:03:16.959 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-22T21:03:16.959 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:03:16.959 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:03:16.959 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-22 21:03:02.441859113 +0000 2024-04-22T21:03:16.959 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-22 20:56:54.292680673 +0000 2024-04-22T21:03:16.959 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-22 20:56:54.292680673 +0000 2024-04-22T21:03:16.960 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-22 20:56:54.292680673 +0000 2024-04-22T21:03:16.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T21:03:17.024 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-22T21:03:17.024 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-22T21:03:17.024 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000290164 s, 1.8 MB/s 2024-04-22T21:03:17.025 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T21:03:17.084 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-04-22T21:03:17.140 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T21:03:17.141 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:03:17.141 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-22T21:03:17.141 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:03:17.141 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:03:17.141 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-22 21:03:02.441859113 +0000 2024-04-22T21:03:17.141 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-22 20:56:54.638675702 +0000 2024-04-22T21:03:17.141 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-22 20:56:54.638675702 +0000 2024-04-22T21:03:17.141 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-22 20:56:54.638675702 +0000 2024-04-22T21:03:17.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T21:03:17.205 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-22T21:03:17.205 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-22T21:03:17.205 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000207932 s, 2.5 MB/s 2024-04-22T21:03:17.206 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T21:03:17.263 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-04-22T21:03:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T21:03:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:03:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-22T21:03:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:03:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:03:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-22 21:03:02.442859100 +0000 2024-04-22T21:03:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-22 20:56:54.990670645 +0000 2024-04-22T21:03:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-22 20:56:54.990670645 +0000 2024-04-22T21:03:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-22 20:56:54.990670645 +0000 2024-04-22T21:03:17.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T21:03:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-22T21:03:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-22T21:03:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000221603 s, 2.3 MB/s 2024-04-22T21:03:17.384 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T21:03:17.441 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-04-22T21:03:17.497 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T21:03:17.497 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:03:17.497 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-22T21:03:17.497 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:03:17.497 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:03:17.497 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-22 21:03:02.442859100 +0000 2024-04-22T21:03:17.497 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-22 20:56:55.334665703 +0000 2024-04-22T21:03:17.498 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-22 20:56:55.334665703 +0000 2024-04-22T21:03:17.498 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-22 20:56:55.334665703 +0000 2024-04-22T21:03:17.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T21:03:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-22T21:03:17.560 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-22T21:03:17.561 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000203773 s, 2.5 MB/s 2024-04-22T21:03:17.561 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T21:03:17.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:17 smithi038 ceph-mon[26374]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:17.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:17 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:17.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:17 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:17.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:17 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi038", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T21:03:17.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:17 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:17.617 INFO:tasks.cephadm:Deploying osd.0 on smithi038 with /dev/vg_nvme/lv_4... 2024-04-22T21:03:17.617 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T21:03:17.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:17 smithi098 ceph-mon[34053]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:17.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:17 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:17.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:17 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:17.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:17 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi038", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T21:03:17.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:17 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:17.887 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:03:18.616 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:18 smithi038 ceph-mon[26374]: Reconfiguring crash.smithi038 (monmap changed)... 2024-04-22T21:03:18.617 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:18 smithi038 ceph-mon[26374]: Reconfiguring daemon crash.smithi038 on smithi038 2024-04-22T21:03:18.617 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:18 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:18.617 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:18 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:18.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:18 smithi098 ceph-mon[34053]: Reconfiguring crash.smithi038 (monmap changed)... 2024-04-22T21:03:18.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:18 smithi098 ceph-mon[34053]: Reconfiguring daemon crash.smithi038 on smithi038 2024-04-22T21:03:18.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:18 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:18.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:18 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:19.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:19 smithi038 ceph-mon[26374]: Reconfiguring alertmanager.smithi038 (dependencies changed)... 2024-04-22T21:03:19.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:19 smithi038 ceph-mon[26374]: Reconfiguring daemon alertmanager.smithi038 on smithi038 2024-04-22T21:03:19.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:19 smithi038 ceph-mon[26374]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:19.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:19 smithi098 ceph-mon[34053]: Reconfiguring alertmanager.smithi038 (dependencies changed)... 2024-04-22T21:03:19.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:19 smithi098 ceph-mon[34053]: Reconfiguring daemon alertmanager.smithi038 on smithi038 2024-04-22T21:03:19.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:19 smithi098 ceph-mon[34053]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:21.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:21 smithi098 ceph-mon[34053]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:21.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:21 smithi038 ceph-mon[26374]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:21.916 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T21:03:21.916 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T21:03:21.916 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-22T21:03:21.916 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-22T21:03:21.916 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144608 s, 725 MB/s 2024-04-22T21:03:21.916 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-22T21:03:23.052 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_4 2024-04-22T21:03:23.351 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:03:23.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:23 smithi038 ceph-mon[26374]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:23.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:23 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:03:23.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:23 smithi098 ceph-mon[34053]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:23.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:23 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:03:25.434 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:25 smithi038 ceph-mon[26374]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:25.434 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:25 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:25.435 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:25 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:25.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.486+0000 7f899525b700 1 -- 172.21.15.38:0/534677703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89900fe040 msgr2=0x7f89900fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:25.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.486+0000 7f899525b700 1 --2- 172.21.15.38:0/534677703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89900fe040 0x7f89900fe4b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8984004940 tx=0x7f8984009e30 comp rx=0 tx=0).stop 2024-04-22T21:03:25.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.486+0000 7f899525b700 1 -- 172.21.15.38:0/534677703 shutdown_connections 2024-04-22T21:03:25.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.486+0000 7f899525b700 1 --2- 172.21.15.38:0/534677703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89900fe040 0x7f89900fe4b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:25.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.486+0000 7f899525b700 1 --2- 172.21.15.38:0/534677703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89900fce40 0x7f89900fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:25.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.486+0000 7f899525b700 1 -- 172.21.15.38:0/534677703 >> 172.21.15.38:0/534677703 conn(0x7f89900f8590 msgr2=0x7f89900fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:03:25.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.486+0000 7f899525b700 1 -- 172.21.15.38:0/534677703 shutdown_connections 2024-04-22T21:03:25.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.486+0000 7f899525b700 1 -- 172.21.15.38:0/534677703 wait complete. 2024-04-22T21:03:25.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.486+0000 7f899525b700 1 Processor -- start 2024-04-22T21:03:25.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.486+0000 7f899525b700 1 -- start start 2024-04-22T21:03:25.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.487+0000 7f899525b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89900fce40 0x7f899010da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:03:25.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.487+0000 7f899525b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89900fe040 0x7f899010dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:03:25.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.487+0000 7f899525b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899010e5a0 con 0x7f89900fe040 2024-04-22T21:03:25.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.487+0000 7f899525b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899010edf0 con 0x7f89900fce40 2024-04-22T21:03:25.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.487+0000 7f898e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89900fe040 0x7f899010dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:03:25.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.487+0000 7f898e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89900fe040 0x7f899010dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36914/0 (socket says 172.21.15.38:36914) 2024-04-22T21:03:25.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.487+0000 7f898e59c700 1 -- 172.21.15.38:0/3199239668 learned_addr learned my addr 172.21.15.38:0/3199239668 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:03:25.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.487+0000 7f898ed9d700 1 --2- 172.21.15.38:0/3199239668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89900fce40 0x7f899010da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:03:25.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.487+0000 7f898e59c700 1 -- 172.21.15.38:0/3199239668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89900fce40 msgr2=0x7f899010da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:25.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.487+0000 7f898e59c700 1 --2- 172.21.15.38:0/3199239668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89900fce40 0x7f899010da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:25.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.487+0000 7f898e59c700 1 -- 172.21.15.38:0/3199239668 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8984005040 con 0x7f89900fe040 2024-04-22T21:03:25.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.487+0000 7f898e59c700 1 --2- 172.21.15.38:0/3199239668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89900fe040 0x7f899010dfa0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8984009150 tx=0x7f8984009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:03:25.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.487+0000 7f89777fe700 1 -- 172.21.15.38:0/3199239668 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f898400e040 con 0x7f89900fe040 2024-04-22T21:03:25.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.488+0000 7f89777fe700 1 -- 172.21.15.38:0/3199239668 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8984004290 con 0x7f89900fe040 2024-04-22T21:03:25.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.488+0000 7f899525b700 1 -- 172.21.15.38:0/3199239668 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f899010f070 con 0x7f89900fe040 2024-04-22T21:03:25.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.488+0000 7f89777fe700 1 -- 172.21.15.38:0/3199239668 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f898401c630 con 0x7f89900fe040 2024-04-22T21:03:25.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.488+0000 7f899525b700 1 -- 172.21.15.38:0/3199239668 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f899010f5b0 con 0x7f89900fe040 2024-04-22T21:03:25.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.489+0000 7f899525b700 1 -- 172.21.15.38:0/3199239668 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f899005f5f0 con 0x7f89900fe040 2024-04-22T21:03:25.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.489+0000 7f89777fe700 1 -- 172.21.15.38:0/3199239668 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7f898400bbf0 con 0x7f89900fe040 2024-04-22T21:03:25.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.490+0000 7f89777fe700 1 --2- 172.21.15.38:0/3199239668 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f897806d620 0x7f897806fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:03:25.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.490+0000 7f89777fe700 1 -- 172.21.15.38:0/3199239668 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f8984086c50 con 0x7f89900fe040 2024-04-22T21:03:25.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.490+0000 7f898ed9d700 1 --2- 172.21.15.38:0/3199239668 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f897806d620 0x7f897806fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:03:25.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.490+0000 7f898ed9d700 1 --2- 172.21.15.38:0/3199239668 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f897806d620 0x7f897806fad0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f89900fdbb0 tx=0x7f898000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:03:25.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.494+0000 7f89777fe700 1 -- 172.21.15.38:0/3199239668 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f898408c020 con 0x7f89900fe040 2024-04-22T21:03:25.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:25.702+0000 7f899525b700 1 -- 172.21.15.38:0/3199239668 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f899010f8e0 con 0x7f897806d620 2024-04-22T21:03:25.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:25 smithi098 ceph-mon[34053]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:25.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:25 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:25.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:25 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:26.500 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:26 smithi038 ceph-mon[26374]: Reconfiguring grafana.smithi038 (dependencies changed)... 2024-04-22T21:03:26.500 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:26 smithi038 ceph-mon[26374]: Reconfiguring daemon grafana.smithi038 on smithi038 2024-04-22T21:03:26.500 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:03:26.500 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:03:26.500 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:26.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:26 smithi098 ceph-mon[34053]: Reconfiguring grafana.smithi038 (dependencies changed)... 2024-04-22T21:03:26.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:26 smithi098 ceph-mon[34053]: Reconfiguring daemon grafana.smithi038 on smithi038 2024-04-22T21:03:26.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:03:26.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:03:26.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:27.528 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:27 smithi038 ceph-mon[26374]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:03:27.529 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:27 smithi038 ceph-mon[26374]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:27.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:27 smithi098 ceph-mon[34053]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:03:27.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:27 smithi098 ceph-mon[34053]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:28.473 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:28 smithi038 ceph-mon[26374]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:28.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:28 smithi098 ceph-mon[34053]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:31.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:30 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:31.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:30 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:31.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:30 smithi038 ceph-mon[26374]: Reconfiguring prometheus.smithi038 (dependencies changed)... 2024-04-22T21:03:31.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:30 smithi038 ceph-mon[26374]: Reconfiguring daemon prometheus.smithi038 on smithi038 2024-04-22T21:03:31.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:30 smithi038 ceph-mon[26374]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:31.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:30 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:31.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:30 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:31.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:30 smithi098 ceph-mon[34053]: Reconfiguring prometheus.smithi038 (dependencies changed)... 2024-04-22T21:03:31.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:30 smithi098 ceph-mon[34053]: Reconfiguring daemon prometheus.smithi038 on smithi038 2024-04-22T21:03:31.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:30 smithi098 ceph-mon[34053]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:32.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:32 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1910380560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aeb7d8ce-e081-4a6a-8c2e-560bcb819a2f"}]: dispatch 2024-04-22T21:03:32.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:32 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1910380560' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aeb7d8ce-e081-4a6a-8c2e-560bcb819a2f"}]': finished 2024-04-22T21:03:32.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:32 smithi038 ceph-mon[26374]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T21:03:32.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:32 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:03:32.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:32 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/2285374374' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:03:32.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:32 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/1910380560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aeb7d8ce-e081-4a6a-8c2e-560bcb819a2f"}]: dispatch 2024-04-22T21:03:32.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:32 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/1910380560' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aeb7d8ce-e081-4a6a-8c2e-560bcb819a2f"}]': finished 2024-04-22T21:03:32.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:32 smithi098 ceph-mon[34053]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T21:03:32.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:32 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:03:32.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:32 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/2285374374' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:03:33.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:33 smithi038 ceph-mon[26374]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:33.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:33 smithi098 ceph-mon[34053]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:35.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:35 smithi038 ceph-mon[26374]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:35.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:35 smithi098 ceph-mon[34053]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:37.353 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:37 smithi038 ceph-mon[26374]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:37.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:37 smithi098 ceph-mon[34053]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:38.216 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:03:38.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:38 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:03:39.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:39 smithi038 ceph-mon[26374]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:39.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:39 smithi098 ceph-mon[34053]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:41.213 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:41 smithi038 ceph-mon[26374]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:41.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:41 smithi098 ceph-mon[34053]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:42.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:42 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:42.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:42 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:42.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:42 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi098", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T21:03:42.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:42 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:42.641 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:42 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:42.641 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:42 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:42.641 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:42 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi098", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T21:03:42.641 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:42 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:43.411 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:43 smithi098 ceph-mon[34053]: Reconfiguring crash.smithi098 (monmap changed)... 2024-04-22T21:03:43.411 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:43 smithi098 ceph-mon[34053]: Reconfiguring daemon crash.smithi098 on smithi098 2024-04-22T21:03:43.412 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:43 smithi098 ceph-mon[34053]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:43.412 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:43 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:43.412 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:43 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:43.412 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:43 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi098.azqohf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:03:43.412 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:43 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:03:43.412 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:43 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:43.412 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:43 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T21:03:43.412 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:43 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:43.500 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:43 smithi038 ceph-mon[26374]: Reconfiguring crash.smithi098 (monmap changed)... 2024-04-22T21:03:43.500 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:43 smithi038 ceph-mon[26374]: Reconfiguring daemon crash.smithi098 on smithi098 2024-04-22T21:03:43.500 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:43 smithi038 ceph-mon[26374]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:43.501 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:43 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:43.501 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:43 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:43.501 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:43 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi098.azqohf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:03:43.501 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:43 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:03:43.501 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:43 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:43.501 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:43 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T21:03:43.501 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:43 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:44.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:44 smithi098 ceph-mon[34053]: Reconfiguring mgr.smithi098.azqohf (monmap changed)... 2024-04-22T21:03:44.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:44 smithi098 ceph-mon[34053]: Reconfiguring daemon mgr.smithi098.azqohf on smithi098 2024-04-22T21:03:44.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:44 smithi098 ceph-mon[34053]: Deploying daemon osd.0 on smithi038 2024-04-22T21:03:44.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:44 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:44.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:44 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:44.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:44 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:03:44.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:44 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T21:03:44.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:44 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:44.606 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:44 smithi038 ceph-mon[26374]: Reconfiguring mgr.smithi098.azqohf (monmap changed)... 2024-04-22T21:03:44.606 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:44 smithi038 ceph-mon[26374]: Reconfiguring daemon mgr.smithi098.azqohf on smithi098 2024-04-22T21:03:44.606 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:44 smithi038 ceph-mon[26374]: Deploying daemon osd.0 on smithi038 2024-04-22T21:03:44.606 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:44 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:44.606 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:44 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:44.607 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:44 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:03:44.607 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:44 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T21:03:44.607 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:44 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:45.494 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:45 smithi038 ceph-mon[26374]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:45.494 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:45 smithi038 ceph-mon[26374]: Reconfiguring mon.smithi098 (monmap changed)... 2024-04-22T21:03:45.494 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:45 smithi038 ceph-mon[26374]: Reconfiguring daemon mon.smithi098 on smithi098 2024-04-22T21:03:45.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:45 smithi098 ceph-mon[34053]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:45.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:45 smithi098 ceph-mon[34053]: Reconfiguring mon.smithi098 (monmap changed)... 2024-04-22T21:03:45.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:45 smithi098 ceph-mon[34053]: Reconfiguring daemon mon.smithi098 on smithi098 2024-04-22T21:03:46.835 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:46.836 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:46.836 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T21:03:46.836 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T21:03:46.836 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi038:9093"}]: dispatch 2024-04-22T21:03:46.836 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi038:9093"}]: dispatch 2024-04-22T21:03:46.836 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:46.836 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T21:03:46.837 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T21:03:46.837 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi038:3000"}]: dispatch 2024-04-22T21:03:46.837 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi038:3000"}]: dispatch 2024-04-22T21:03:46.837 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:46.837 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T21:03:46.837 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T21:03:46.837 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi038:9095"}]: dispatch 2024-04-22T21:03:46.838 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi038:9095"}]: dispatch 2024-04-22T21:03:46.838 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:46.838 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:03:46.838 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:46 smithi038 ceph-mon[26374]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:46.852 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:46.852 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:46.852 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T21:03:46.852 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T21:03:46.852 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi038:9093"}]: dispatch 2024-04-22T21:03:46.852 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi038:9093"}]: dispatch 2024-04-22T21:03:46.852 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:46.852 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T21:03:46.852 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T21:03:46.853 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi038:3000"}]: dispatch 2024-04-22T21:03:46.853 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi038:3000"}]: dispatch 2024-04-22T21:03:46.853 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:46.853 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T21:03:46.853 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T21:03:46.853 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi038:9095"}]: dispatch 2024-04-22T21:03:46.853 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi038:9095"}]: dispatch 2024-04-22T21:03:46.853 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:46.853 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:03:46.853 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:46 smithi098 ceph-mon[34053]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:48.602 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:48.602 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:48.603 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:48 smithi038 ceph-mon[26374]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:48.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:48.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:48.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:48 smithi098 ceph-mon[34053]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:50.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:49 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:50.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:49 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:50.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:49 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:50.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:49 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:51.113 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:50 smithi038 ceph-mon[26374]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:51.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:50 smithi098 ceph-mon[34053]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:53.330 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:53.330 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:53 smithi098 ceph-mon[34053]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:53.330 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:53.330 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:53.331 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:03:53.331 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:53.331 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:03:53.331 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:53 smithi098 ceph-mon[34053]: from='osd.0 [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T21:03:53.355 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:53.356 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:53 smithi038 ceph-mon[26374]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:53.356 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:53.356 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:03:53.356 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:03:53.356 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:53.356 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:03:53.356 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:53 smithi038 ceph-mon[26374]: from='osd.0 [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T21:03:53.437 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 0 on host 'smithi038' 2024-04-22T21:03:53.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:53.440+0000 7f89777fe700 1 -- 172.21.15.38:0/3199239668 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f899010f8e0 con 0x7f897806d620 2024-04-22T21:03:53.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:53.443+0000 7f899525b700 1 -- 172.21.15.38:0/3199239668 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f897806d620 msgr2=0x7f897806fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:53.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:53.443+0000 7f899525b700 1 --2- 172.21.15.38:0/3199239668 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f897806d620 0x7f897806fad0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f89900fdbb0 tx=0x7f898000b040 comp rx=0 tx=0).stop 2024-04-22T21:03:53.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:53.443+0000 7f899525b700 1 -- 172.21.15.38:0/3199239668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89900fe040 msgr2=0x7f899010dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:03:53.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:53.443+0000 7f899525b700 1 --2- 172.21.15.38:0/3199239668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89900fe040 0x7f899010dfa0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8984009150 tx=0x7f8984009e30 comp rx=0 tx=0).stop 2024-04-22T21:03:53.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:53.443+0000 7f899525b700 1 -- 172.21.15.38:0/3199239668 shutdown_connections 2024-04-22T21:03:53.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:53.443+0000 7f899525b700 1 --2- 172.21.15.38:0/3199239668 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f897806d620 0x7f897806fad0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:53.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:53.443+0000 7f899525b700 1 --2- 172.21.15.38:0/3199239668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89900fce40 0x7f899010da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:53.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:53.443+0000 7f899525b700 1 --2- 172.21.15.38:0/3199239668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89900fe040 0x7f899010dfa0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:03:53.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:53.443+0000 7f899525b700 1 -- 172.21.15.38:0/3199239668 >> 172.21.15.38:0/3199239668 conn(0x7f89900f8590 msgr2=0x7f89900f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:03:53.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:53.443+0000 7f899525b700 1 -- 172.21.15.38:0/3199239668 shutdown_connections 2024-04-22T21:03:53.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:03:53.443+0000 7f899525b700 1 -- 172.21.15.38:0/3199239668 wait complete. 2024-04-22T21:03:54.041 DEBUG:teuthology.orchestra.run.smithi038:osd.0> sudo journalctl -f -n 0 -u ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@osd.0.service 2024-04-22T21:03:54.043 INFO:tasks.cephadm:Deploying osd.1 on smithi038 with /dev/vg_nvme/lv_3... 2024-04-22T21:03:54.044 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T21:03:54.290 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:54 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:03:54.290 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:54 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:54.290 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:54 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:54.291 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:54 smithi038 ceph-mon[26374]: from='osd.0 [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T21:03:54.291 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:54 smithi038 ceph-mon[26374]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T21:03:54.291 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:54 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:03:54.291 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:54 smithi038 ceph-mon[26374]: from='osd.0 [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T21:03:54.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:54 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:03:54.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:54 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:54.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:54 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:54.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:54 smithi098 ceph-mon[34053]: from='osd.0 [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T21:03:54.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:54 smithi098 ceph-mon[34053]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T21:03:54.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:54 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:03:54.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:54 smithi098 ceph-mon[34053]: from='osd.0 [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T21:03:54.697 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:03:54.933 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 21:03:54 smithi038 ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-osd-0[45038]: 2024-04-22T21:03:54.685+0000 7f1fbcd33700 -1 osd.0 0 waiting for initial osdmap 2024-04-22T21:03:54.933 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 21:03:54 smithi038 ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-osd-0[45038]: 2024-04-22T21:03:54.690+0000 7f1fb5b2a700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:03:55.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:55 smithi038 ceph-mon[26374]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:55.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:55 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:55.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:55 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:55.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:55 smithi038 ceph-mon[26374]: from='osd.0 [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T21:03:55.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:55 smithi038 ceph-mon[26374]: osdmap e8: 1 total, 0 up, 1 in 2024-04-22T21:03:55.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:55 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:03:55.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:55 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:03:55.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:55 smithi098 ceph-mon[34053]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:55.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:55 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:55.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:55 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:03:55.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:55 smithi098 ceph-mon[34053]: from='osd.0 [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T21:03:55.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:55 smithi098 ceph-mon[34053]: osdmap e8: 1 total, 0 up, 1 in 2024-04-22T21:03:55.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:55 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:03:55.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:55 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:03:56.523 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T21:03:56.523 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T21:03:56.523 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-22T21:03:56.523 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-22T21:03:56.523 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.018012 s, 582 MB/s 2024-04-22T21:03:56.524 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-22T21:03:56.987 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:56 smithi038 ceph-mon[26374]: purged_snaps scrub starts 2024-04-22T21:03:56.987 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:56 smithi038 ceph-mon[26374]: purged_snaps scrub ok 2024-04-22T21:03:56.987 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:56 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:03:56.987 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:56 smithi038 ceph-mon[26374]: osd.0 [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338] boot 2024-04-22T21:03:56.987 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:56 smithi038 ceph-mon[26374]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T21:03:56.987 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:56 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:03:56.987 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:56 smithi038 ceph-mon[26374]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:57.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:56 smithi098 ceph-mon[34053]: purged_snaps scrub starts 2024-04-22T21:03:57.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:56 smithi098 ceph-mon[34053]: purged_snaps scrub ok 2024-04-22T21:03:57.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:56 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:03:57.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:56 smithi098 ceph-mon[34053]: osd.0 [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338] boot 2024-04-22T21:03:57.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:56 smithi098 ceph-mon[34053]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T21:03:57.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:56 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:03:57.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:56 smithi098 ceph-mon[34053]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:03:57.281 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_3 2024-04-22T21:03:57.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:57 smithi038 ceph-mon[26374]: osdmap e10: 1 total, 1 up, 1 in 2024-04-22T21:03:58.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:57 smithi098 ceph-mon[34053]: osdmap e10: 1 total, 1 up, 1 in 2024-04-22T21:03:58.071 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:03:58.996 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:03:58 smithi038 ceph-mon[26374]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:03:58.999 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:03:58 smithi098 ceph-mon[34053]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:00.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.471+0000 7f3fad3d9700 1 -- 172.21.15.38:0/3578082864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fa80fcfb0 msgr2=0x7f3fa80fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:04:00.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.471+0000 7f3fad3d9700 1 --2- 172.21.15.38:0/3578082864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fa80fcfb0 0x7f3fa80fd420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c004940 tx=0x7f3f9c009e30 comp rx=0 tx=0).stop 2024-04-22T21:04:00.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.471+0000 7f3fad3d9700 1 -- 172.21.15.38:0/3578082864 shutdown_connections 2024-04-22T21:04:00.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.471+0000 7f3fad3d9700 1 --2- 172.21.15.38:0/3578082864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fa80fcfb0 0x7f3fa80fd420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:04:00.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.471+0000 7f3fad3d9700 1 --2- 172.21.15.38:0/3578082864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa80fbd00 0x7f3fa80fc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:04:00.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.471+0000 7f3fad3d9700 1 -- 172.21.15.38:0/3578082864 >> 172.21.15.38:0/3578082864 conn(0x7f3fa80f7480 msgr2=0x7f3fa80f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:04:00.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.471+0000 7f3fad3d9700 1 -- 172.21.15.38:0/3578082864 shutdown_connections 2024-04-22T21:04:00.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.471+0000 7f3fad3d9700 1 -- 172.21.15.38:0/3578082864 wait complete. 2024-04-22T21:04:00.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.472+0000 7f3fad3d9700 1 Processor -- start 2024-04-22T21:04:00.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.472+0000 7f3fad3d9700 1 -- start start 2024-04-22T21:04:00.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.472+0000 7f3fad3d9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa80fbd00 0x7f3fa810c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:04:00.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.472+0000 7f3fad3d9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fa80fcfb0 0x7f3fa810ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:04:00.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.472+0000 7f3fad3d9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa810d490 con 0x7f3fa80fcfb0 2024-04-22T21:04:00.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.472+0000 7f3fad3d9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa8111e30 con 0x7f3fa80fbd00 2024-04-22T21:04:00.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.473+0000 7f3fa67fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fa80fcfb0 0x7f3fa810ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:04:00.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.473+0000 7f3fa67fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fa80fcfb0 0x7f3fa810ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49348/0 (socket says 172.21.15.38:49348) 2024-04-22T21:04:00.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.473+0000 7f3fa67fc700 1 -- 172.21.15.38:0/3778745134 learned_addr learned my addr 172.21.15.38:0/3778745134 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:04:00.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.473+0000 7f3fa6ffd700 1 --2- 172.21.15.38:0/3778745134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa80fbd00 0x7f3fa810c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:04:00.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.473+0000 7f3fa67fc700 1 -- 172.21.15.38:0/3778745134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa80fbd00 msgr2=0x7f3fa810c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:04:00.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.473+0000 7f3fa67fc700 1 --2- 172.21.15.38:0/3778745134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa80fbd00 0x7f3fa810c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:04:00.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.473+0000 7f3fa67fc700 1 -- 172.21.15.38:0/3778745134 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9c005040 con 0x7f3fa80fcfb0 2024-04-22T21:04:00.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.473+0000 7f3fa67fc700 1 --2- 172.21.15.38:0/3778745134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fa80fcfb0 0x7f3fa810ce00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c0090d0 tx=0x7f3f9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:04:00.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.473+0000 7f3f8f7fe700 1 -- 172.21.15.38:0/3778745134 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f9c00e040 con 0x7f3fa80fcfb0 2024-04-22T21:04:00.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.473+0000 7f3f8f7fe700 1 -- 172.21.15.38:0/3778745134 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f9c004290 con 0x7f3fa80fcfb0 2024-04-22T21:04:00.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.473+0000 7f3fad3d9700 1 -- 172.21.15.38:0/3778745134 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa8112030 con 0x7f3fa80fcfb0 2024-04-22T21:04:00.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.473+0000 7f3f8f7fe700 1 -- 172.21.15.38:0/3778745134 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f9c01c3f0 con 0x7f3fa80fcfb0 2024-04-22T21:04:00.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.473+0000 7f3fad3d9700 1 -- 172.21.15.38:0/3778745134 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa81125a0 con 0x7f3fa80fcfb0 2024-04-22T21:04:00.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.474+0000 7f3fad3d9700 1 -- 172.21.15.38:0/3778745134 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa804c710 con 0x7f3fa80fcfb0 2024-04-22T21:04:00.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.474+0000 7f3f8f7fe700 1 -- 172.21.15.38:0/3778745134 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7f3f9c00bbf0 con 0x7f3fa80fcfb0 2024-04-22T21:04:00.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.475+0000 7f3f8f7fe700 1 --2- 172.21.15.38:0/3778745134 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f3f90069210 0x7f3f9006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:04:00.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.475+0000 7f3f8f7fe700 1 -- 172.21.15.38:0/3778745134 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f3f9c087530 con 0x7f3fa80fcfb0 2024-04-22T21:04:00.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.475+0000 7f3fa6ffd700 1 --2- 172.21.15.38:0/3778745134 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f3f90069210 0x7f3f9006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:04:00.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.475+0000 7f3fa6ffd700 1 --2- 172.21.15.38:0/3778745134 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f3f90069210 0x7f3f9006b6c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3f98006fd0 tx=0x7f3f98008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:04:00.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.480+0000 7f3f8f7fe700 1 -- 172.21.15.38:0/3778745134 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f3f9c0582c0 con 0x7f3fa80fcfb0 2024-04-22T21:04:00.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:00.684+0000 7f3fad3d9700 1 -- 172.21.15.38:0/3778745134 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fa81012d0 con 0x7f3f90069210 2024-04-22T21:04:01.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:01 smithi038 ceph-mon[26374]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:01.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:01 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:04:01.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:01 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:04:01.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:01 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:01.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:01 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:01.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:01 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:01.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:01 smithi098 ceph-mon[34053]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:01.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:01 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:04:01.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:01 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:04:01.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:01 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:01.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:01 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:01.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:01 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:02.266 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:02 smithi038 ceph-mon[26374]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:04:02.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:02 smithi098 ceph-mon[34053]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:04:03.319 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:03 smithi038 ceph-mon[26374]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:03.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:03 smithi098 ceph-mon[34053]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:04.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:04 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3083526788' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c90956f-6cb0-4a71-86c0-9c38e9710808"}]: dispatch 2024-04-22T21:04:04.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:04 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3083526788' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c90956f-6cb0-4a71-86c0-9c38e9710808"}]': finished 2024-04-22T21:04:04.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:04 smithi038 ceph-mon[26374]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T21:04:04.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:04 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:04:04.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:04 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/2252639607' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:04:04.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/3083526788' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c90956f-6cb0-4a71-86c0-9c38e9710808"}]: dispatch 2024-04-22T21:04:04.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/3083526788' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c90956f-6cb0-4a71-86c0-9c38e9710808"}]': finished 2024-04-22T21:04:04.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:04 smithi098 ceph-mon[34053]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T21:04:04.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:04:04.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/2252639607' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:04:05.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:05 smithi038 ceph-mon[26374]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:05.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:05 smithi098 ceph-mon[34053]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:07.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:07 smithi038 ceph-mon[26374]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:07.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:07 smithi098 ceph-mon[34053]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:08.549 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:08 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:04:08.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:08 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:04:09.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:09 smithi098 ceph-mon[34053]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:09.556 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:09 smithi038 ceph-mon[26374]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:11.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:11 smithi038 ceph-mon[26374]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:11.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:11 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:11.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:11 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:11.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:11 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:04:11.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:11 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:11.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:11 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:11.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:11 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:04:11.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:11 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:11.626 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:11 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:04:11.644 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:11 smithi098 ceph-mon[34053]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:11.645 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:11 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:11.645 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:11 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:11.645 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:11 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:04:11.645 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:11 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:11.645 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:11 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:11.645 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:11 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:04:11.645 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:11 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:11.645 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:11 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:04:12.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:12 smithi038 ceph-mon[26374]: Detected new or changed devices on smithi038 2024-04-22T21:04:12.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:12 smithi038 ceph-mon[26374]: Adjusting osd_memory_target on smithi038 to 12869M 2024-04-22T21:04:12.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:12 smithi098 ceph-mon[34053]: Detected new or changed devices on smithi038 2024-04-22T21:04:12.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:12 smithi098 ceph-mon[34053]: Adjusting osd_memory_target on smithi038 to 12869M 2024-04-22T21:04:13.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:13 smithi038 ceph-mon[26374]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:13.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T21:04:13.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:13.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:13 smithi038 ceph-mon[26374]: Deploying daemon osd.1 on smithi038 2024-04-22T21:04:13.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:13.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:13.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:13 smithi098 ceph-mon[34053]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:13.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T21:04:13.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:13.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:13 smithi098 ceph-mon[34053]: Deploying daemon osd.1 on smithi038 2024-04-22T21:04:13.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:13.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:14.647 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:14 smithi038 ceph-mon[26374]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:14.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:14 smithi098 ceph-mon[34053]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:17.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:17 smithi038 ceph-mon[26374]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:17.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:17 smithi098 ceph-mon[34053]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:19.366 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:19 smithi038 ceph-mon[26374]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:19.367 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:19 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:19.367 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:19 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:19.367 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:19 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:19.367 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:19 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:04:19.367 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:19 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:19.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:19 smithi098 ceph-mon[34053]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:19.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:19 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:19.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:19 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:19.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:19 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:19.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:19 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:04:19.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:19 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:20.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:20 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:20.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:20 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:20.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:20 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:04:20.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:20 smithi038 ceph-mon[26374]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:20.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:20 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:20.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:20 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:20.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:20 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:04:20.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:20 smithi098 ceph-mon[34053]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:22.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:22.962+0000 7f3f8f7fe700 1 -- 172.21.15.38:0/3778745134 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3fa81012d0 con 0x7f3f90069210 2024-04-22T21:04:22.959 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 1 on host 'smithi038' 2024-04-22T21:04:22.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:22.964+0000 7f3fad3d9700 1 -- 172.21.15.38:0/3778745134 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f3f90069210 msgr2=0x7f3f9006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:04:22.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:22.964+0000 7f3fad3d9700 1 --2- 172.21.15.38:0/3778745134 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f3f90069210 0x7f3f9006b6c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3f98006fd0 tx=0x7f3f98008040 comp rx=0 tx=0).stop 2024-04-22T21:04:22.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:22.964+0000 7f3fad3d9700 1 -- 172.21.15.38:0/3778745134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fa80fcfb0 msgr2=0x7f3fa810ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:04:22.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:22.964+0000 7f3fad3d9700 1 --2- 172.21.15.38:0/3778745134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fa80fcfb0 0x7f3fa810ce00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c0090d0 tx=0x7f3f9c009e30 comp rx=0 tx=0).stop 2024-04-22T21:04:22.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:22.964+0000 7f3fad3d9700 1 -- 172.21.15.38:0/3778745134 shutdown_connections 2024-04-22T21:04:22.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:22.964+0000 7f3fad3d9700 1 --2- 172.21.15.38:0/3778745134 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f3f90069210 0x7f3f9006b6c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:04:22.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:22.964+0000 7f3fad3d9700 1 --2- 172.21.15.38:0/3778745134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa80fbd00 0x7f3fa810c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:04:22.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:22.964+0000 7f3fad3d9700 1 --2- 172.21.15.38:0/3778745134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fa80fcfb0 0x7f3fa810ce00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:04:22.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:22.964+0000 7f3fad3d9700 1 -- 172.21.15.38:0/3778745134 >> 172.21.15.38:0/3778745134 conn(0x7f3fa80f7480 msgr2=0x7f3fa80f8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:04:22.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:22.964+0000 7f3fad3d9700 1 -- 172.21.15.38:0/3778745134 shutdown_connections 2024-04-22T21:04:22.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:22.964+0000 7f3fad3d9700 1 -- 172.21.15.38:0/3778745134 wait complete. 2024-04-22T21:04:23.291 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:23 smithi038 ceph-mon[26374]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:23.291 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:23 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:23.291 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:23 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:23.291 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:23 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:04:23.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:23 smithi098 ceph-mon[34053]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:23.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:23 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:23.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:23 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:23.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:23 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:04:24.210 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:24 smithi038 ceph-mon[26374]: from='osd.1 [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T21:04:24.235 DEBUG:teuthology.orchestra.run.smithi038:osd.1> sudo journalctl -f -n 0 -u ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@osd.1.service 2024-04-22T21:04:24.238 INFO:tasks.cephadm:Deploying osd.2 on smithi038 with /dev/vg_nvme/lv_2... 2024-04-22T21:04:24.238 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T21:04:24.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:24 smithi098 ceph-mon[34053]: from='osd.1 [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T21:04:24.624 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:04:25.249 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:25 smithi038 ceph-mon[26374]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:25.249 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:25 smithi038 ceph-mon[26374]: from='osd.1 [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T21:04:25.249 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:25 smithi038 ceph-mon[26374]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T21:04:25.249 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:25 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:04:25.249 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:25 smithi038 ceph-mon[26374]: from='osd.1 [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T21:04:25.250 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 22 21:04:25 smithi038 ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-osd-1[52953]: 2024-04-22T21:04:25.128+0000 7fd6768f2700 -1 osd.1 0 waiting for initial osdmap 2024-04-22T21:04:25.250 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 22 21:04:25 smithi038 ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-osd-1[52953]: 2024-04-22T21:04:25.136+0000 7fd6736f1700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:04:25.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:25 smithi098 ceph-mon[34053]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:25.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:25 smithi098 ceph-mon[34053]: from='osd.1 [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T21:04:25.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:25 smithi098 ceph-mon[34053]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T21:04:25.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:25 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:04:25.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:25 smithi098 ceph-mon[34053]: from='osd.1 [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T21:04:26.330 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:26 smithi038 ceph-mon[26374]: from='osd.1 [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T21:04:26.330 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:26 smithi038 ceph-mon[26374]: osdmap e13: 2 total, 1 up, 2 in 2024-04-22T21:04:26.330 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:04:26.330 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:04:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:26 smithi098 ceph-mon[34053]: from='osd.1 [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T21:04:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:26 smithi098 ceph-mon[34053]: osdmap e13: 2 total, 1 up, 2 in 2024-04-22T21:04:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:04:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:04:27.323 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:27 smithi038 ceph-mon[26374]: purged_snaps scrub starts 2024-04-22T21:04:27.323 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:27 smithi038 ceph-mon[26374]: purged_snaps scrub ok 2024-04-22T21:04:27.323 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:27 smithi038 ceph-mon[26374]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:27.323 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:27 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:04:27.323 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:27 smithi038 ceph-mon[26374]: osd.1 [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940] boot 2024-04-22T21:04:27.323 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:27 smithi038 ceph-mon[26374]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T21:04:27.323 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:27 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:04:27.324 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:27 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:27.324 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:27 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:27.324 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T21:04:27.324 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T21:04:27.324 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-22T21:04:27.324 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-22T21:04:27.324 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00954354 s, 1.1 GB/s 2024-04-22T21:04:27.324 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-22T21:04:27.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:27 smithi098 ceph-mon[34053]: purged_snaps scrub starts 2024-04-22T21:04:27.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:27 smithi098 ceph-mon[34053]: purged_snaps scrub ok 2024-04-22T21:04:27.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:27 smithi098 ceph-mon[34053]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:04:27.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:27 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:04:27.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:27 smithi098 ceph-mon[34053]: osd.1 [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940] boot 2024-04-22T21:04:27.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:27 smithi098 ceph-mon[34053]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T21:04:27.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:27 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:04:27.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:27 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:27.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:27 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:28.086 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_2 2024-04-22T21:04:28.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:28 smithi038 ceph-mon[26374]: osdmap e15: 2 total, 2 up, 2 in 2024-04-22T21:04:28.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:28 smithi098 ceph-mon[34053]: osdmap e15: 2 total, 2 up, 2 in 2024-04-22T21:04:29.314 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:29 smithi038 ceph-mon[26374]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:29.369 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:04:29.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:29 smithi098 ceph-mon[34053]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:31.348 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:31 smithi038 ceph-mon[26374]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:31.348 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:31 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:31.348 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:31 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:31.348 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:31 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:04:31.348 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:31 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:04:31.348 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:31 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:31.348 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:31 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:31.348 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:31 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:04:31.348 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:31 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:31.348 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:31 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:04:31.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:31 smithi098 ceph-mon[34053]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:31.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:31 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:31.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:31 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:31.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:31 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:04:31.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:31 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:04:31.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:31 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:31.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:31 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:31.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:31 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:04:31.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:31 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:31.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:31 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:04:31.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.578+0000 7f889ebf7700 1 -- 172.21.15.38:0/151013120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8898066b20 msgr2=0x7f88980ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:04:31.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.578+0000 7f889ebf7700 1 --2- 172.21.15.38:0/151013120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8898066b20 0x7f88980ffa40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8890009ce0 tx=0x7f8890004d10 comp rx=0 tx=0).stop 2024-04-22T21:04:31.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.578+0000 7f889ebf7700 1 -- 172.21.15.38:0/151013120 shutdown_connections 2024-04-22T21:04:31.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.578+0000 7f889ebf7700 1 --2- 172.21.15.38:0/151013120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88980fff80 0x7f8898102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:04:31.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.578+0000 7f889ebf7700 1 --2- 172.21.15.38:0/151013120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8898066b20 0x7f88980ffa40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:04:31.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.578+0000 7f889ebf7700 1 -- 172.21.15.38:0/151013120 >> 172.21.15.38:0/151013120 conn(0x7f88980f7450 msgr2=0x7f88980f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:04:31.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.578+0000 7f889ebf7700 1 -- 172.21.15.38:0/151013120 shutdown_connections 2024-04-22T21:04:31.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.578+0000 7f889ebf7700 1 -- 172.21.15.38:0/151013120 wait complete. 2024-04-22T21:04:31.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.578+0000 7f889ebf7700 1 Processor -- start 2024-04-22T21:04:31.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.579+0000 7f889ebf7700 1 -- start start 2024-04-22T21:04:31.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.579+0000 7f889ebf7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8898066b20 0x7f889810dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:04:31.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.579+0000 7f889ebf7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88980fff80 0x7f889810e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:04:31.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.579+0000 7f889ebf7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889810e740 con 0x7f88980fff80 2024-04-22T21:04:31.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.579+0000 7f889ebf7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889810ef90 con 0x7f8898066b20 2024-04-22T21:04:31.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.579+0000 7f8897fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88980fff80 0x7f889810e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:04:31.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.579+0000 7f8897fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88980fff80 0x7f889810e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60552/0 (socket says 172.21.15.38:60552) 2024-04-22T21:04:31.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.579+0000 7f8897fff700 1 -- 172.21.15.38:0/3087376893 learned_addr learned my addr 172.21.15.38:0/3087376893 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:04:31.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.579+0000 7f889c993700 1 --2- 172.21.15.38:0/3087376893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8898066b20 0x7f889810dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:04:31.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.579+0000 7f8897fff700 1 -- 172.21.15.38:0/3087376893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8898066b20 msgr2=0x7f889810dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:04:31.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.579+0000 7f8897fff700 1 --2- 172.21.15.38:0/3087376893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8898066b20 0x7f889810dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:04:31.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.579+0000 7f8897fff700 1 -- 172.21.15.38:0/3087376893 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8890009940 con 0x7f88980fff80 2024-04-22T21:04:31.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.579+0000 7f8897fff700 1 --2- 172.21.15.38:0/3087376893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88980fff80 0x7f889810e140 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f888c002a50 tx=0x7f888c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:04:31.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.580+0000 7f88957fa700 1 -- 172.21.15.38:0/3087376893 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f888c007cc0 con 0x7f88980fff80 2024-04-22T21:04:31.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.580+0000 7f889ebf7700 1 -- 172.21.15.38:0/3087376893 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889810f270 con 0x7f88980fff80 2024-04-22T21:04:31.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.580+0000 7f88957fa700 1 -- 172.21.15.38:0/3087376893 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f888c00f040 con 0x7f88980fff80 2024-04-22T21:04:31.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.580+0000 7f88957fa700 1 -- 172.21.15.38:0/3087376893 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f888c00da30 con 0x7f88980fff80 2024-04-22T21:04:31.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.580+0000 7f889ebf7700 1 -- 172.21.15.38:0/3087376893 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889810f7b0 con 0x7f88980fff80 2024-04-22T21:04:31.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.581+0000 7f889ebf7700 1 -- 172.21.15.38:0/3087376893 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f889805f5f0 con 0x7f88980fff80 2024-04-22T21:04:31.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.581+0000 7f88957fa700 1 -- 172.21.15.38:0/3087376893 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7f888c00db90 con 0x7f88980fff80 2024-04-22T21:04:31.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.581+0000 7f88957fa700 1 --2- 172.21.15.38:0/3087376893 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8880069210 0x7f888006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:04:31.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.582+0000 7f88957fa700 1 -- 172.21.15.38:0/3087376893 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f888c086240 con 0x7f88980fff80 2024-04-22T21:04:31.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.582+0000 7f889c993700 1 --2- 172.21.15.38:0/3087376893 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8880069210 0x7f888006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:04:31.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.582+0000 7f889c993700 1 --2- 172.21.15.38:0/3087376893 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8880069210 0x7f888006b6c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8890009470 tx=0x7f8890004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:04:31.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.586+0000 7f88957fa700 1 -- 172.21.15.38:0/3087376893 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f888c056c50 con 0x7f88980fff80 2024-04-22T21:04:31.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:31.795+0000 7f889ebf7700 1 -- 172.21.15.38:0/3087376893 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f889810fae0 con 0x7f8880069210 2024-04-22T21:04:32.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:32 smithi038 ceph-mon[26374]: Adjusting osd_memory_target on smithi038 to 6434M 2024-04-22T21:04:32.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:32 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:04:32.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:32 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:04:32.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:32 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:32.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:32 smithi098 ceph-mon[34053]: Adjusting osd_memory_target on smithi038 to 6434M 2024-04-22T21:04:32.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:32 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:04:32.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:32 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:04:32.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:32 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:33.362 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:33 smithi038 ceph-mon[26374]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:04:33.363 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:33 smithi038 ceph-mon[26374]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:33.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:33 smithi098 ceph-mon[34053]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:04:33.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:33 smithi098 ceph-mon[34053]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:35.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:35 smithi038 ceph-mon[26374]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:35.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:35 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/280265715' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "379616b9-bdc7-4fd6-a1c5-b937e2ef3202"}]: dispatch 2024-04-22T21:04:35.469 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:35 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/280265715' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "379616b9-bdc7-4fd6-a1c5-b937e2ef3202"}]': finished 2024-04-22T21:04:35.469 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:35 smithi038 ceph-mon[26374]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T21:04:35.469 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:35 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:04:35.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:35 smithi098 ceph-mon[34053]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:35.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:35 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/280265715' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "379616b9-bdc7-4fd6-a1c5-b937e2ef3202"}]: dispatch 2024-04-22T21:04:35.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:35 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/280265715' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "379616b9-bdc7-4fd6-a1c5-b937e2ef3202"}]': finished 2024-04-22T21:04:35.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:35 smithi098 ceph-mon[34053]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T21:04:35.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:35 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:04:36.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:36 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/4239884657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:04:36.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:36 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/4239884657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:04:37.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:37 smithi098 ceph-mon[34053]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:37.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:37 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:37.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:37 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:37.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:37 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:37.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:37 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:04:37.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:37 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:37.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:37 smithi038 ceph-mon[26374]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:37.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:37 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:37.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:37 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:37.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:37 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:37.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:37 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:04:37.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:37 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:38.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:38 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:04:38.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:04:39.495 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:39 smithi038 ceph-mon[26374]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:39.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:39 smithi098 ceph-mon[34053]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:41.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:41 smithi038 ceph-mon[26374]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:41.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:41 smithi098 ceph-mon[34053]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:43.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:43 smithi038 ceph-mon[26374]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:43.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:43 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T21:04:43.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:43 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:43.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:43 smithi098 ceph-mon[34053]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:43.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:43 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T21:04:43.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:43 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:44.285 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:44 smithi038 ceph-mon[26374]: Deploying daemon osd.2 on smithi038 2024-04-22T21:04:44.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:44 smithi098 ceph-mon[34053]: Deploying daemon osd.2 on smithi038 2024-04-22T21:04:45.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:45 smithi038 ceph-mon[26374]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:45.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:45 smithi098 ceph-mon[34053]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:47.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:47 smithi038 ceph-mon[26374]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:47.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:47 smithi098 ceph-mon[34053]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:48.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:04:48.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:48.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:48.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:04:48.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:48.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:49.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:49 smithi038 ceph-mon[26374]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:49.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:49 smithi098 ceph-mon[34053]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:51.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:51 smithi098 ceph-mon[34053]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:51.535 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:51 smithi038 ceph-mon[26374]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:51.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:51.771+0000 7f88957fa700 1 -- 172.21.15.38:0/3087376893 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f889810fae0 con 0x7f8880069210 2024-04-22T21:04:51.768 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 2 on host 'smithi038' 2024-04-22T21:04:51.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:51.773+0000 7f889ebf7700 1 -- 172.21.15.38:0/3087376893 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8880069210 msgr2=0x7f888006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:04:51.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:51.773+0000 7f889ebf7700 1 --2- 172.21.15.38:0/3087376893 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8880069210 0x7f888006b6c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8890009470 tx=0x7f8890004d10 comp rx=0 tx=0).stop 2024-04-22T21:04:51.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:51.773+0000 7f889ebf7700 1 -- 172.21.15.38:0/3087376893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88980fff80 msgr2=0x7f889810e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:04:51.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:51.773+0000 7f889ebf7700 1 --2- 172.21.15.38:0/3087376893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88980fff80 0x7f889810e140 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f888c002a50 tx=0x7f888c0046b0 comp rx=0 tx=0).stop 2024-04-22T21:04:51.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:51.774+0000 7f889ebf7700 1 -- 172.21.15.38:0/3087376893 shutdown_connections 2024-04-22T21:04:51.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:51.774+0000 7f889ebf7700 1 --2- 172.21.15.38:0/3087376893 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8880069210 0x7f888006b6c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:04:51.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:51.774+0000 7f889ebf7700 1 --2- 172.21.15.38:0/3087376893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8898066b20 0x7f889810dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:04:51.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:51.774+0000 7f889ebf7700 1 --2- 172.21.15.38:0/3087376893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88980fff80 0x7f889810e140 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:04:51.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:51.774+0000 7f889ebf7700 1 -- 172.21.15.38:0/3087376893 >> 172.21.15.38:0/3087376893 conn(0x7f88980f7450 msgr2=0x7f88980f7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:04:51.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:51.774+0000 7f889ebf7700 1 -- 172.21.15.38:0/3087376893 shutdown_connections 2024-04-22T21:04:51.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:04:51.774+0000 7f889ebf7700 1 -- 172.21.15.38:0/3087376893 wait complete. 2024-04-22T21:04:52.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:52 smithi038 ceph-mon[26374]: from='osd.2 [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T21:04:52.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:52 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:52.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:52 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:52.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:52 smithi098 ceph-mon[34053]: from='osd.2 [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T21:04:52.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:52 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:52.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:52 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:53.033 DEBUG:teuthology.orchestra.run.smithi038:osd.2> sudo journalctl -f -n 0 -u ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@osd.2.service 2024-04-22T21:04:53.035 INFO:tasks.cephadm:Deploying osd.3 on smithi098 with /dev/vg_nvme/lv_4... 2024-04-22T21:04:53.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T21:04:53.295 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi098/config 2024-04-22T21:04:53.416 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:53 smithi098 ceph-mon[34053]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:53.417 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:53 smithi098 ceph-mon[34053]: from='osd.2 [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T21:04:53.417 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:53 smithi098 ceph-mon[34053]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T21:04:53.417 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:04:53.417 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:53 smithi098 ceph-mon[34053]: from='osd.2 [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T21:04:53.417 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:04:53.515 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:53 smithi038 ceph-mon[26374]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:53.515 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:53 smithi038 ceph-mon[26374]: from='osd.2 [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T21:04:53.515 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:53 smithi038 ceph-mon[26374]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T21:04:53.515 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:04:53.515 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:53 smithi038 ceph-mon[26374]: from='osd.2 [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T21:04:53.515 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:04:53.516 INFO:journalctl@ceph.osd.2.smithi038.stdout:Apr 22 21:04:53 smithi038 ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-osd-2[60286]: 2024-04-22T21:04:53.262+0000 7f16d2a2c700 -1 osd.2 0 waiting for initial osdmap 2024-04-22T21:04:53.516 INFO:journalctl@ceph.osd.2.smithi038.stdout:Apr 22 21:04:53 smithi038 ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-osd-2[60286]: 2024-04-22T21:04:53.271+0000 7f16ce028700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:04:54.590 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:54 smithi098 ceph-mon[34053]: from='osd.2 [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T21:04:54.590 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:54 smithi098 ceph-mon[34053]: osdmap e18: 3 total, 2 up, 3 in 2024-04-22T21:04:54.590 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:54 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:04:54.590 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:54 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:04:54.590 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:54 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:54.590 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:54 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:54.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:54 smithi038 ceph-mon[26374]: from='osd.2 [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T21:04:54.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:54 smithi038 ceph-mon[26374]: osdmap e18: 3 total, 2 up, 3 in 2024-04-22T21:04:54.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:54 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:04:54.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:54 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:04:54.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:54 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:54.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:54 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:55.142 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T21:04:55.142 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T21:04:55.142 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-22T21:04:55.142 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-22T21:04:55.143 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011343 s, 924 MB/s 2024-04-22T21:04:55.143 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-22T21:04:55.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:55 smithi038 ceph-mon[26374]: purged_snaps scrub starts 2024-04-22T21:04:55.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:55 smithi038 ceph-mon[26374]: purged_snaps scrub ok 2024-04-22T21:04:55.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:55 smithi038 ceph-mon[26374]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:55.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:55 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:04:55.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:55 smithi038 ceph-mon[26374]: osd.2 [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714] boot 2024-04-22T21:04:55.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:55 smithi038 ceph-mon[26374]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T21:04:55.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:55 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:04:55.527 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:55 smithi098 ceph-mon[34053]: purged_snaps scrub starts 2024-04-22T21:04:55.527 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:55 smithi098 ceph-mon[34053]: purged_snaps scrub ok 2024-04-22T21:04:55.527 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:55 smithi098 ceph-mon[34053]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-22T21:04:55.528 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:55 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:04:55.528 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:55 smithi098 ceph-mon[34053]: osd.2 [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714] boot 2024-04-22T21:04:55.528 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:55 smithi098 ceph-mon[34053]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T21:04:55.528 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:55 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:04:55.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2024-04-22T21:04:56.088 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi098/config 2024-04-22T21:04:56.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:56 smithi038 ceph-mon[26374]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T21:04:56.885 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:56 smithi098 ceph-mon[34053]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T21:04:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.705+0000 7fb0ccd4b700 1 -- 172.21.15.98:0/3227893664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c8071e40 msgr2=0x7fb0c8070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:04:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.705+0000 7fb0ccd4b700 1 --2- 172.21.15.98:0/3227893664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c8071e40 0x7fb0c8070350 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc004970 tx=0x7fb0bc009e30 comp rx=0 tx=0).stop 2024-04-22T21:04:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.705+0000 7fb0ccd4b700 1 -- 172.21.15.98:0/3227893664 shutdown_connections 2024-04-22T21:04:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.705+0000 7fb0ccd4b700 1 --2- 172.21.15.98:0/3227893664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0c8070890 0x7fb0c8070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:04:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.705+0000 7fb0ccd4b700 1 --2- 172.21.15.98:0/3227893664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c8071e40 0x7fb0c8070350 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:04:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.705+0000 7fb0ccd4b700 1 -- 172.21.15.98:0/3227893664 >> 172.21.15.98:0/3227893664 conn(0x7fb0c80f85e0 msgr2=0x7fb0c80fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:04:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.705+0000 7fb0ccd4b700 1 -- 172.21.15.98:0/3227893664 shutdown_connections 2024-04-22T21:04:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.705+0000 7fb0ccd4b700 1 -- 172.21.15.98:0/3227893664 wait complete. 2024-04-22T21:04:57.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.706+0000 7fb0ccd4b700 1 Processor -- start 2024-04-22T21:04:57.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.706+0000 7fb0ccd4b700 1 -- start start 2024-04-22T21:04:57.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.706+0000 7fb0ccd4b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0c8070890 0x7fb0c80fcd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:04:57.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.706+0000 7fb0ccd4b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c8071e40 0x7fb0c80fd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:04:57.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.706+0000 7fb0ccd4b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c80fd9a0 con 0x7fb0c8070890 2024-04-22T21:04:57.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.706+0000 7fb0ccd4b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c80fe110 con 0x7fb0c8071e40 2024-04-22T21:04:57.703 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: pgmap v77: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-22T21:04:57.703 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-22T21:04:57.703 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-22T21:04:57.704 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T21:04:57.704 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T21:04:57.704 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: Detected new or changed devices on smithi038 2024-04-22T21:04:57.704 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:57.704 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:57.704 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:04:57.704 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:04:57.704 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:04:57.704 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: Adjusting osd_memory_target on smithi038 to 4289M 2024-04-22T21:04:57.704 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:57.704 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:57.704 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:04:57.704 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:57.704 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:57 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:04:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.707+0000 7fb0c5d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c8071e40 0x7fb0c80fd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:04:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.707+0000 7fb0c5d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c8071e40 0x7fb0c80fd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37040/0 (socket says 172.21.15.98:37040) 2024-04-22T21:04:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.707+0000 7fb0c5d9b700 1 -- 172.21.15.98:0/363107652 learned_addr learned my addr 172.21.15.98:0/363107652 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:04:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.707+0000 7fb0c659c700 1 --2- 172.21.15.98:0/363107652 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0c8070890 0x7fb0c80fcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:04:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.707+0000 7fb0c5d9b700 1 -- 172.21.15.98:0/363107652 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0c8070890 msgr2=0x7fb0c80fcd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:04:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.707+0000 7fb0c5d9b700 1 --2- 172.21.15.98:0/363107652 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0c8070890 0x7fb0c80fcd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:04:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.707+0000 7fb0c5d9b700 1 -- 172.21.15.98:0/363107652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0bc005040 con 0x7fb0c8071e40 2024-04-22T21:04:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.707+0000 7fb0c5d9b700 1 --2- 172.21.15.98:0/363107652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c8071e40 0x7fb0c80fd2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800b790 tx=0x7fb0b8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:04:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.707+0000 7fb0b6ffd700 1 -- 172.21.15.98:0/363107652 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0b800f6f0 con 0x7fb0c8071e40 2024-04-22T21:04:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.707+0000 7fb0ccd4b700 1 -- 172.21.15.98:0/363107652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c80fe3f0 con 0x7fb0c8071e40 2024-04-22T21:04:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.707+0000 7fb0ccd4b700 1 -- 172.21.15.98:0/363107652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c80fe900 con 0x7fb0c8071e40 2024-04-22T21:04:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.708+0000 7fb0b6ffd700 1 -- 172.21.15.98:0/363107652 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0b800fd20 con 0x7fb0c8071e40 2024-04-22T21:04:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.708+0000 7fb0b6ffd700 1 -- 172.21.15.98:0/363107652 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0b80103a0 con 0x7fb0c8071e40 2024-04-22T21:04:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.709+0000 7fb0ccd4b700 1 -- 172.21.15.98:0/363107652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c805f5f0 con 0x7fb0c8071e40 2024-04-22T21:04:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.709+0000 7fb0b6ffd700 1 -- 172.21.15.98:0/363107652 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7fb0b800f850 con 0x7fb0c8071e40 2024-04-22T21:04:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.710+0000 7fb0b6ffd700 1 --2- 172.21.15.98:0/363107652 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fb0b0069210 0x7fb0b006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:04:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.710+0000 7fb0b6ffd700 1 -- 172.21.15.98:0/363107652 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3228+0+0 (secure 0 0 0) 0x7fb0b80864a0 con 0x7fb0c8071e40 2024-04-22T21:04:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.710+0000 7fb0c659c700 1 --2- 172.21.15.98:0/363107652 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fb0b0069210 0x7fb0b006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:04:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.710+0000 7fb0c659c700 1 --2- 172.21.15.98:0/363107652 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fb0b0069210 0x7fb0b006b6c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009fd0 tx=0x7fb0bc009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:04:57.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.716+0000 7fb0b6ffd700 1 -- 172.21.15.98:0/363107652 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb0b8053210 con 0x7fb0c8071e40 2024-04-22T21:04:57.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: pgmap v77: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-22T21:04:57.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-22T21:04:57.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-22T21:04:57.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T21:04:57.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T21:04:57.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: Detected new or changed devices on smithi038 2024-04-22T21:04:57.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:57.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:57.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:04:57.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:04:57.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:04:57.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: Adjusting osd_memory_target on smithi038 to 4289M 2024-04-22T21:04:57.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:57.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:57.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:04:57.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:57.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:57 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:04:57.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:04:57.913+0000 7fb0ccd4b700 1 -- 172.21.15.98:0/363107652 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0c80fec40 con 0x7fb0b0069210 2024-04-22T21:04:58.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:58 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T21:04:58.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:58 smithi038 ceph-mon[26374]: osdmap e22: 3 total, 3 up, 3 in 2024-04-22T21:04:58.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:58 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:58.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:58 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:04:58.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:58 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:58.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:58 smithi038 ceph-mon[26374]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:04:58.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:58 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:04:58.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:58 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:04:58.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:58 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:58.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:58 smithi038 ceph-mon[26374]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-22T21:04:58.954 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:58 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T21:04:58.954 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:58 smithi098 ceph-mon[34053]: osdmap e22: 3 total, 3 up, 3 in 2024-04-22T21:04:58.954 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:58 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:58.954 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:58 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:04:58.954 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:58 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:04:58.955 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:58 smithi098 ceph-mon[34053]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:04:58.955 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:58 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:04:58.955 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:58 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:04:58.955 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:58 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:04:58.955 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:58 smithi098 ceph-mon[34053]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-22T21:04:59.374 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 21:04:59 smithi038 sudo[64710]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T21:04:59.375 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 21:04:59 smithi038 sudo[64710]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T21:04:59.375 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 21:04:59 smithi038 sudo[64710]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T21:04:59.375 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 21:04:59 smithi038 sudo[64710]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:04:59.868 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 21:04:59 smithi038 sudo[64713]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T21:04:59.868 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 21:04:59 smithi038 sudo[64713]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T21:04:59.868 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 21:04:59 smithi038 sudo[64713]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T21:04:59.868 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 21:04:59 smithi038 sudo[64713]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:05:00.124 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:59 smithi038 ceph-mon[26374]: osdmap e23: 3 total, 3 up, 3 in 2024-04-22T21:05:00.124 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:04:59 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:00.125 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 22 21:04:59 smithi038 sudo[64716]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T21:05:00.125 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 22 21:04:59 smithi038 sudo[64716]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T21:05:00.125 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 22 21:04:59 smithi038 sudo[64716]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T21:05:00.125 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 22 21:04:59 smithi038 sudo[64716]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:05:00.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:59 smithi098 ceph-mon[34053]: osdmap e23: 3 total, 3 up, 3 in 2024-04-22T21:05:00.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:04:59 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:00.827 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 22 21:05:00 smithi038 sudo[64719]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T21:05:00.827 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 22 21:05:00 smithi038 sudo[64719]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T21:05:00.827 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 22 21:05:00 smithi038 sudo[64719]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T21:05:00.827 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 22 21:05:00 smithi038 sudo[64719]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:05:00.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:00.883+0000 7fb0b6ffd700 1 -- 172.21.15.98:0/363107652 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fb0b8011400 con 0x7fb0c8071e40 2024-04-22T21:05:01.125 INFO:journalctl@ceph.osd.2.smithi038.stdout:Apr 22 21:05:00 smithi038 sudo[64722]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T21:05:01.125 INFO:journalctl@ceph.osd.2.smithi038.stdout:Apr 22 21:05:00 smithi038 sudo[64722]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T21:05:01.125 INFO:journalctl@ceph.osd.2.smithi038.stdout:Apr 22 21:05:00 smithi038 sudo[64722]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T21:05:01.125 INFO:journalctl@ceph.osd.2.smithi038.stdout:Apr 22 21:05:00 smithi038 sudo[64722]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:05:01.126 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:00 smithi038 ceph-mon[26374]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-22T21:05:01.126 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:00 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/2678215520' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "becb1602-d471-4c38-8cd1-781828c4b2f7"}]: dispatch 2024-04-22T21:05:01.126 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:00 smithi038 ceph-mon[26374]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "becb1602-d471-4c38-8cd1-781828c4b2f7"}]: dispatch 2024-04-22T21:05:01.126 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:00 smithi038 ceph-mon[26374]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "becb1602-d471-4c38-8cd1-781828c4b2f7"}]': finished 2024-04-22T21:05:01.127 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:00 smithi038 ceph-mon[26374]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T21:05:01.127 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:00 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:05:01.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:00 smithi098 ceph-mon[34053]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-22T21:05:01.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:00 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/2678215520' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "becb1602-d471-4c38-8cd1-781828c4b2f7"}]: dispatch 2024-04-22T21:05:01.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:00 smithi098 ceph-mon[34053]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "becb1602-d471-4c38-8cd1-781828c4b2f7"}]: dispatch 2024-04-22T21:05:01.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:00 smithi098 ceph-mon[34053]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "becb1602-d471-4c38-8cd1-781828c4b2f7"}]': finished 2024-04-22T21:05:01.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:00 smithi098 ceph-mon[34053]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T21:05:01.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:00 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:05:01.535 INFO:journalctl@ceph.osd.2.smithi038.stdout:Apr 22 21:05:01 smithi038 sudo[64725]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T21:05:01.536 INFO:journalctl@ceph.osd.2.smithi038.stdout:Apr 22 21:05:01 smithi038 sudo[64725]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T21:05:01.536 INFO:journalctl@ceph.osd.2.smithi038.stdout:Apr 22 21:05:01 smithi038 sudo[64725]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T21:05:01.536 INFO:journalctl@ceph.osd.2.smithi038.stdout:Apr 22 21:05:01 smithi038 sudo[64725]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:05:01.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:01 smithi038 sudo[64728]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T21:05:01.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:01 smithi038 sudo[64728]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T21:05:01.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:01 smithi038 sudo[64728]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T21:05:01.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:01 smithi038 sudo[64728]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:05:02.453 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:02 smithi098 ceph-mon[34053]: mgrmap e19: smithi038.lxfgut(active, since 2m), standbys: smithi098.azqohf 2024-04-22T21:05:02.453 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:02 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/129758388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:05:02.453 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:02 smithi098 ceph-mon[34053]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:05:02.454 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:02 smithi098 sudo[38046]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T21:05:02.454 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:02 smithi098 sudo[38046]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T21:05:02.454 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:02 smithi098 sudo[38046]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T21:05:02.454 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:02 smithi098 sudo[38046]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:05:02.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:02 smithi038 sudo[64731]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T21:05:02.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:02 smithi038 sudo[64731]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T21:05:02.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:02 smithi038 sudo[64731]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T21:05:02.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:02 smithi038 sudo[64731]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:05:02.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:02 smithi038 ceph-mon[26374]: mgrmap e19: smithi038.lxfgut(active, since 2m), standbys: smithi098.azqohf 2024-04-22T21:05:02.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:02 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/129758388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:05:02.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:02 smithi038 ceph-mon[26374]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:05:02.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:02 smithi098 sudo[38235]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T21:05:02.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:02 smithi098 sudo[38235]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T21:05:02.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:02 smithi098 sudo[38235]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T21:05:02.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:02 smithi098 sudo[38235]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:05:03.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:03 smithi098 ceph-mon[34053]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-04-22T21:05:03.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:03 smithi098 ceph-mon[34053]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:05:03.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:03 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2024-04-22T21:05:03.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:03 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:05:03.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:03 smithi098 ceph-mon[34053]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:05:03.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:03 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2024-04-22T21:05:03.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:03 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:05:03.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:03 smithi098 ceph-mon[34053]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:05:03.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:03 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:03.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:03 smithi038 ceph-mon[26374]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-04-22T21:05:03.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:03 smithi038 ceph-mon[26374]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:05:03.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:03 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2024-04-22T21:05:03.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:03 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:05:03.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:03 smithi038 ceph-mon[26374]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:05:03.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:03 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2024-04-22T21:05:03.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:03 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mon metadata", "id": "smithi098"}]: dispatch 2024-04-22T21:05:03.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:03 smithi038 ceph-mon[26374]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:05:03.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:03 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:05.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:05 smithi098 ceph-mon[34053]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-22T21:05:05.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:05 smithi038 ceph-mon[26374]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-22T21:05:07.452 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:07 smithi098 ceph-mon[34053]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-22T21:05:07.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:07 smithi038 ceph-mon[26374]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-22T21:05:08.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:08 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T21:05:08.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:08 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:08.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:08 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:05:08.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:08 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T21:05:08.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:08 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:08.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:08 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:05:09.454 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:09 smithi098 ceph-mon[34053]: Deploying daemon osd.3 on smithi098 2024-04-22T21:05:09.454 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:09 smithi098 ceph-mon[34053]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-04-22T21:05:09.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:09 smithi038 ceph-mon[26374]: Deploying daemon osd.3 on smithi098 2024-04-22T21:05:09.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:09 smithi038 ceph-mon[26374]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-04-22T21:05:11.368 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:11 smithi098 ceph-mon[34053]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-22T21:05:11.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:11 smithi038 ceph-mon[26374]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-22T21:05:13.224 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:13 smithi098 ceph-mon[34053]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-22T21:05:13.224 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:13.225 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:05:13.225 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:13.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:13 smithi038 ceph-mon[26374]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-22T21:05:13.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:13.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:05:13.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:15.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:15.423+0000 7fb0b6ffd700 1 -- 172.21.15.98:0/363107652 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb0c80fec40 con 0x7fb0b0069210 2024-04-22T21:05:15.422 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 3 on host 'smithi098' 2024-04-22T21:05:15.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:15.424+0000 7fb0ccd4b700 1 -- 172.21.15.98:0/363107652 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fb0b0069210 msgr2=0x7fb0b006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:05:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:15.425+0000 7fb0ccd4b700 1 --2- 172.21.15.98:0/363107652 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fb0b0069210 0x7fb0b006b6c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009fd0 tx=0x7fb0bc009e30 comp rx=0 tx=0).stop 2024-04-22T21:05:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:15.425+0000 7fb0ccd4b700 1 -- 172.21.15.98:0/363107652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c8071e40 msgr2=0x7fb0c80fd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:05:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:15.425+0000 7fb0ccd4b700 1 --2- 172.21.15.98:0/363107652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c8071e40 0x7fb0c80fd2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800b790 tx=0x7fb0b8009250 comp rx=0 tx=0).stop 2024-04-22T21:05:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:15.425+0000 7fb0ccd4b700 1 -- 172.21.15.98:0/363107652 shutdown_connections 2024-04-22T21:05:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:15.425+0000 7fb0ccd4b700 1 --2- 172.21.15.98:0/363107652 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fb0b0069210 0x7fb0b006b6c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:05:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:15.425+0000 7fb0ccd4b700 1 --2- 172.21.15.98:0/363107652 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0c8070890 0x7fb0c80fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:05:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:15.425+0000 7fb0ccd4b700 1 --2- 172.21.15.98:0/363107652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c8071e40 0x7fb0c80fd2c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:05:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:15.425+0000 7fb0ccd4b700 1 -- 172.21.15.98:0/363107652 >> 172.21.15.98:0/363107652 conn(0x7fb0c80f85e0 msgr2=0x7fb0c80f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:05:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:15.425+0000 7fb0ccd4b700 1 -- 172.21.15.98:0/363107652 shutdown_connections 2024-04-22T21:05:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:15.425+0000 7fb0ccd4b700 1 -- 172.21.15.98:0/363107652 wait complete. 2024-04-22T21:05:15.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:15 smithi098 ceph-mon[34053]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-22T21:05:15.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:15 smithi038 ceph-mon[26374]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-22T21:05:16.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:16 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:16.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:16 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:16.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:16 smithi038 ceph-mon[26374]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T21:05:16.686 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:16 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:16.687 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:16 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:16.687 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:16 smithi098 ceph-mon[34053]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T21:05:16.957 DEBUG:teuthology.orchestra.run.smithi098:osd.3> sudo journalctl -f -n 0 -u ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@osd.3.service 2024-04-22T21:05:16.959 INFO:tasks.cephadm:Deploying osd.4 on smithi098 with /dev/vg_nvme/lv_3... 2024-04-22T21:05:16.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T21:05:17.262 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi098/config 2024-04-22T21:05:17.724 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:17 smithi098 ceph-mon[34053]: from='osd.3 [v2:172.21.15.98:6800/2538219727,v1:172.21.15.98:6801/2538219727]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T21:05:17.724 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:17 smithi098 ceph-mon[34053]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T21:05:17.724 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:17 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:17.724 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:17 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:17.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:17 smithi038 ceph-mon[26374]: from='osd.3 [v2:172.21.15.98:6800/2538219727,v1:172.21.15.98:6801/2538219727]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T21:05:17.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:17 smithi038 ceph-mon[26374]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T21:05:17.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:17 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:17.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:17 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:18.717 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:18 smithi098 ceph-mon[34053]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T21:05:18.717 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:18 smithi098 ceph-mon[34053]: osdmap e25: 4 total, 3 up, 4 in 2024-04-22T21:05:18.717 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:18 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:05:18.717 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:18 smithi098 ceph-mon[34053]: from='osd.3 [v2:172.21.15.98:6800/2538219727,v1:172.21.15.98:6801/2538219727]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-22T21:05:18.718 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:18 smithi098 ceph-mon[34053]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-22T21:05:18.718 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:18 smithi098 ceph-mon[34053]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-22T21:05:18.718 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 22 21:05:18 smithi098 ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-osd-3[39664]: 2024-04-22T21:05:18.404+0000 7f1c8bdd8700 -1 osd.3 0 waiting for initial osdmap 2024-04-22T21:05:18.718 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 22 21:05:18 smithi098 ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-osd-3[39664]: 2024-04-22T21:05:18.410+0000 7f1c873d4700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:05:18.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:18 smithi038 ceph-mon[26374]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T21:05:18.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:18 smithi038 ceph-mon[26374]: osdmap e25: 4 total, 3 up, 4 in 2024-04-22T21:05:18.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:18 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:05:18.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:18 smithi038 ceph-mon[26374]: from='osd.3 [v2:172.21.15.98:6800/2538219727,v1:172.21.15.98:6801/2538219727]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-22T21:05:18.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:18 smithi038 ceph-mon[26374]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-22T21:05:18.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:18 smithi038 ceph-mon[26374]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-22T21:05:19.698 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:19 smithi098 ceph-mon[34053]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-22T21:05:19.698 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:19 smithi098 ceph-mon[34053]: osdmap e26: 4 total, 3 up, 4 in 2024-04-22T21:05:19.698 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:19 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:05:19.698 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:19 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:05:19.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:19 smithi038 ceph-mon[26374]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-22T21:05:19.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:19 smithi038 ceph-mon[26374]: osdmap e26: 4 total, 3 up, 4 in 2024-04-22T21:05:19.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:19 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:05:19.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:19 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:05:20.634 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:20 smithi098 ceph-mon[34053]: purged_snaps scrub starts 2024-04-22T21:05:20.634 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:20 smithi098 ceph-mon[34053]: purged_snaps scrub ok 2024-04-22T21:05:20.634 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:20 smithi098 ceph-mon[34053]: osd.3 [v2:172.21.15.98:6800/2538219727,v1:172.21.15.98:6801/2538219727] boot 2024-04-22T21:05:20.634 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:20 smithi098 ceph-mon[34053]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T21:05:20.634 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:20 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:05:20.634 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:20 smithi098 ceph-mon[34053]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:20.634 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:20 smithi098 ceph-mon[34053]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T21:05:20.634 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T21:05:20.635 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T21:05:20.635 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-22T21:05:20.635 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-22T21:05:20.635 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124739 s, 841 MB/s 2024-04-22T21:05:20.635 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 2024-04-22T21:05:20.635 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-22T21:05:20.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:20 smithi038 ceph-mon[26374]: purged_snaps scrub starts 2024-04-22T21:05:20.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:20 smithi038 ceph-mon[26374]: purged_snaps scrub ok 2024-04-22T21:05:20.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:20 smithi038 ceph-mon[26374]: osd.3 [v2:172.21.15.98:6800/2538219727,v1:172.21.15.98:6801/2538219727] boot 2024-04-22T21:05:20.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:20 smithi038 ceph-mon[26374]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T21:05:20.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:20 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:05:20.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:20 smithi038 ceph-mon[26374]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:20.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:20 smithi038 ceph-mon[26374]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T21:05:21.726 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2024-04-22T21:05:21.972 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi098/config 2024-04-22T21:05:22.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:22 smithi038 ceph-mon[26374]: Detected new or changed devices on smithi098 2024-04-22T21:05:22.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:22 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:22.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:22 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:22.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:22 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:05:22.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:22 smithi038 ceph-mon[26374]: Adjusting osd_memory_target on smithi098 to 15936M 2024-04-22T21:05:22.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:22 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:22.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:22 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:22.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:22 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:05:22.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:22 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:22.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:22 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:05:22.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:22 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:22.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:22 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:05:22.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:22 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:22.376 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:22 smithi038 ceph-mon[26374]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T21:05:22.388 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:22 smithi098 ceph-mon[34053]: Detected new or changed devices on smithi098 2024-04-22T21:05:22.389 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:22 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:22.389 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:22 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:22.389 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:22 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:05:22.389 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:22 smithi098 ceph-mon[34053]: Adjusting osd_memory_target on smithi098 to 15936M 2024-04-22T21:05:22.389 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:22 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:22.389 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:22 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:22.389 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:22 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:05:22.389 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:22 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:22.389 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:22 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:05:22.389 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:22 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:22.389 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:22 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:05:22.389 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:22 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:22.389 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:22 smithi098 ceph-mon[34053]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T21:05:23.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:23 smithi098 ceph-mon[34053]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:23.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:23 smithi098 ceph-mon[34053]: osdmap e30: 4 total, 4 up, 4 in 2024-04-22T21:05:23.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:23 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:05:23.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.535+0000 7ff71d920700 1 -- 172.21.15.98:0/3815512595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7180fce80 msgr2=0x7ff7180fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:05:23.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.535+0000 7ff71d920700 1 --2- 172.21.15.98:0/3815512595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7180fce80 0x7ff7180fd290 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff708004940 tx=0x7ff708009e30 comp rx=0 tx=0).stop 2024-04-22T21:05:23.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.535+0000 7ff71d920700 1 -- 172.21.15.98:0/3815512595 shutdown_connections 2024-04-22T21:05:23.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.535+0000 7ff71d920700 1 --2- 172.21.15.98:0/3815512595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7180fdff0 0x7ff7180fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:05:23.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.535+0000 7ff71d920700 1 --2- 172.21.15.98:0/3815512595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7180fce80 0x7ff7180fd290 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:05:23.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.535+0000 7ff71d920700 1 -- 172.21.15.98:0/3815512595 >> 172.21.15.98:0/3815512595 conn(0x7ff7180f8540 msgr2=0x7ff7180fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:05:23.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.535+0000 7ff71d920700 1 -- 172.21.15.98:0/3815512595 shutdown_connections 2024-04-22T21:05:23.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.535+0000 7ff71d920700 1 -- 172.21.15.98:0/3815512595 wait complete. 2024-04-22T21:05:23.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.536+0000 7ff71d920700 1 Processor -- start 2024-04-22T21:05:23.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.536+0000 7ff71d920700 1 -- start start 2024-04-22T21:05:23.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.536+0000 7ff71d920700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7180fce80 0x7ff71810d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:05:23.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.536+0000 7ff71d920700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7180fdff0 0x7ff71810deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:05:23.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.536+0000 7ff71d920700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71810e4b0 con 0x7ff7180fce80 2024-04-22T21:05:23.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.536+0000 7ff71d920700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71810e620 con 0x7ff7180fdff0 2024-04-22T21:05:23.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.536+0000 7ff7167fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7180fdff0 0x7ff71810deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:05:23.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.537+0000 7ff7167fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7180fdff0 0x7ff71810deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55032/0 (socket says 172.21.15.98:55032) 2024-04-22T21:05:23.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.537+0000 7ff7167fc700 1 -- 172.21.15.98:0/3043771090 learned_addr learned my addr 172.21.15.98:0/3043771090 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:05:23.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.537+0000 7ff716ffd700 1 --2- 172.21.15.98:0/3043771090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7180fce80 0x7ff71810d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:05:23.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.537+0000 7ff7167fc700 1 -- 172.21.15.98:0/3043771090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7180fce80 msgr2=0x7ff71810d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:05:23.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.537+0000 7ff7167fc700 1 --2- 172.21.15.98:0/3043771090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7180fce80 0x7ff71810d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:05:23.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.537+0000 7ff7167fc700 1 -- 172.21.15.98:0/3043771090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff708005040 con 0x7ff7180fdff0 2024-04-22T21:05:23.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.537+0000 7ff7167fc700 1 --2- 172.21.15.98:0/3043771090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7180fdff0 0x7ff71810deb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff70c002a50 tx=0x7ff70c0046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:05:23.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.537+0000 7ff6ff7fe700 1 -- 172.21.15.98:0/3043771090 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff70c007cc0 con 0x7ff7180fdff0 2024-04-22T21:05:23.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.537+0000 7ff71d920700 1 -- 172.21.15.98:0/3043771090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff718113080 con 0x7ff7180fdff0 2024-04-22T21:05:23.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.537+0000 7ff6ff7fe700 1 -- 172.21.15.98:0/3043771090 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff70c00f040 con 0x7ff7180fdff0 2024-04-22T21:05:23.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.537+0000 7ff6ff7fe700 1 -- 172.21.15.98:0/3043771090 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff70c00da30 con 0x7ff7180fdff0 2024-04-22T21:05:23.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.537+0000 7ff71d920700 1 -- 172.21.15.98:0/3043771090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff718113640 con 0x7ff7180fdff0 2024-04-22T21:05:23.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.538+0000 7ff71d920700 1 -- 172.21.15.98:0/3043771090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71804c710 con 0x7ff7180fdff0 2024-04-22T21:05:23.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.538+0000 7ff6ff7fe700 1 -- 172.21.15.98:0/3043771090 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ff70c00db90 con 0x7ff7180fdff0 2024-04-22T21:05:23.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.539+0000 7ff6ff7fe700 1 --2- 172.21.15.98:0/3043771090 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff7000692a0 0x7ff70006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:05:23.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.539+0000 7ff6ff7fe700 1 -- 172.21.15.98:0/3043771090 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3770+0+0 (secure 0 0 0) 0x7ff70c0868b0 con 0x7ff7180fdff0 2024-04-22T21:05:23.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.539+0000 7ff716ffd700 1 --2- 172.21.15.98:0/3043771090 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff7000692a0 0x7ff70006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:05:23.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.540+0000 7ff716ffd700 1 --2- 172.21.15.98:0/3043771090 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff7000692a0 0x7ff70006b750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff70800b970 tx=0x7ff708009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:05:23.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.543+0000 7ff6ff7fe700 1 -- 172.21.15.98:0/3043771090 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff70c056c90 con 0x7ff7180fdff0 2024-04-22T21:05:23.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:23 smithi038 ceph-mon[26374]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:23.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:23 smithi038 ceph-mon[26374]: osdmap e30: 4 total, 4 up, 4 in 2024-04-22T21:05:23.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:23 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:05:23.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:23.733+0000 7ff71d920700 1 -- 172.21.15.98:0/3043771090 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ff718102400 con 0x7ff7000692a0 2024-04-22T21:05:24.448 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:24 smithi098 ceph-mon[34053]: osdmap e31: 4 total, 4 up, 4 in 2024-04-22T21:05:24.448 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:24 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:05:24.448 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:24 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:05:24.448 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:24 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:24.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:24 smithi038 ceph-mon[26374]: osdmap e31: 4 total, 4 up, 4 in 2024-04-22T21:05:24.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:24 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:05:24.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:24 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:05:24.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:24 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:25.342 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:25 smithi098 ceph-mon[34053]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:05:25.342 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:25 smithi098 ceph-mon[34053]: pgmap v102: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T21:05:25.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:25 smithi038 ceph-mon[26374]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:05:25.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:25 smithi038 ceph-mon[26374]: pgmap v102: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T21:05:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:26 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/4047229627' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c886eb15-63f1-4bb9-a229-0278c8372f5b"}]: dispatch 2024-04-22T21:05:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:26 smithi098 ceph-mon[34053]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c886eb15-63f1-4bb9-a229-0278c8372f5b"}]: dispatch 2024-04-22T21:05:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:26 smithi098 ceph-mon[34053]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c886eb15-63f1-4bb9-a229-0278c8372f5b"}]': finished 2024-04-22T21:05:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:26 smithi098 ceph-mon[34053]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T21:05:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:05:26.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:26 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/4047229627' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c886eb15-63f1-4bb9-a229-0278c8372f5b"}]: dispatch 2024-04-22T21:05:26.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:26 smithi038 ceph-mon[26374]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c886eb15-63f1-4bb9-a229-0278c8372f5b"}]: dispatch 2024-04-22T21:05:26.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:26 smithi038 ceph-mon[26374]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c886eb15-63f1-4bb9-a229-0278c8372f5b"}]': finished 2024-04-22T21:05:26.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:26 smithi038 ceph-mon[26374]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T21:05:26.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:05:27.502 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:27 smithi098 ceph-mon[34053]: pgmap v103: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-22T21:05:27.502 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:27 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/2861036483' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:05:27.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:27 smithi038 ceph-mon[26374]: pgmap v103: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-22T21:05:27.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:27 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/2861036483' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:05:29.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:29 smithi098 ceph-mon[34053]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T21:05:29.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:29 smithi038 ceph-mon[26374]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T21:05:31.424 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:31 smithi098 ceph-mon[34053]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:31.535 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:31 smithi038 ceph-mon[26374]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:33.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:33 smithi098 ceph-mon[34053]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:33.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:33 smithi038 ceph-mon[26374]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:34.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:34 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T21:05:34.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:34 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:34.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:34 smithi098 ceph-mon[34053]: Deploying daemon osd.4 on smithi098 2024-04-22T21:05:34.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:34 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T21:05:34.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:34 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:34.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:34 smithi038 ceph-mon[26374]: Deploying daemon osd.4 on smithi098 2024-04-22T21:05:35.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:35 smithi038 ceph-mon[26374]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:35.713 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:35 smithi098 ceph-mon[34053]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:37.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:37 smithi098 ceph-mon[34053]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:37.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:37 smithi038 ceph-mon[26374]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:38.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:05:38.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:38 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:05:39.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:39 smithi038 ceph-mon[26374]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:39.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:39 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:39.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:39 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:39.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:39 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:05:39.649 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:39 smithi098 ceph-mon[34053]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:39.649 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:39 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:39.649 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:39 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:39.650 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:39 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:05:41.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:41 smithi098 ceph-mon[34053]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:41.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:41 smithi038 ceph-mon[26374]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:41.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:41.999+0000 7ff6ff7fe700 1 -- 172.21.15.98:0/3043771090 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff718102400 con 0x7ff7000692a0 2024-04-22T21:05:41.996 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 4 on host 'smithi098' 2024-04-22T21:05:41.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:42.002+0000 7ff71d920700 1 -- 172.21.15.98:0/3043771090 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff7000692a0 msgr2=0x7ff70006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:05:41.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:42.002+0000 7ff71d920700 1 --2- 172.21.15.98:0/3043771090 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff7000692a0 0x7ff70006b750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff70800b970 tx=0x7ff708009e30 comp rx=0 tx=0).stop 2024-04-22T21:05:41.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:42.002+0000 7ff71d920700 1 -- 172.21.15.98:0/3043771090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7180fdff0 msgr2=0x7ff71810deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:05:41.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:42.002+0000 7ff71d920700 1 --2- 172.21.15.98:0/3043771090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7180fdff0 0x7ff71810deb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff70c002a50 tx=0x7ff70c0046b0 comp rx=0 tx=0).stop 2024-04-22T21:05:41.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:42.002+0000 7ff71d920700 1 -- 172.21.15.98:0/3043771090 shutdown_connections 2024-04-22T21:05:41.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:42.002+0000 7ff71d920700 1 --2- 172.21.15.98:0/3043771090 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff7000692a0 0x7ff70006b750 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:05:41.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:42.002+0000 7ff71d920700 1 --2- 172.21.15.98:0/3043771090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7180fce80 0x7ff71810d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:05:41.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:42.002+0000 7ff71d920700 1 --2- 172.21.15.98:0/3043771090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7180fdff0 0x7ff71810deb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:05:41.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:42.002+0000 7ff71d920700 1 -- 172.21.15.98:0/3043771090 >> 172.21.15.98:0/3043771090 conn(0x7ff7180f8540 msgr2=0x7ff7180f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:05:41.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:42.002+0000 7ff71d920700 1 -- 172.21.15.98:0/3043771090 shutdown_connections 2024-04-22T21:05:41.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:42.002+0000 7ff71d920700 1 -- 172.21.15.98:0/3043771090 wait complete. 2024-04-22T21:05:42.926 DEBUG:teuthology.orchestra.run.smithi098:osd.4> sudo journalctl -f -n 0 -u ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@osd.4.service 2024-04-22T21:05:42.929 INFO:tasks.cephadm:Deploying osd.5 on smithi098 with /dev/vg_nvme/lv_2... 2024-04-22T21:05:42.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T21:05:43.225 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi098/config 2024-04-22T21:05:43.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:42 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:43.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:42 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:43.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:42 smithi098 ceph-mon[34053]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:43.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:42 smithi098 ceph-mon[34053]: from='osd.4 [v2:172.21.15.98:6808/3945015359,v1:172.21.15.98:6809/3945015359]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T21:05:43.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:42 smithi098 ceph-mon[34053]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T21:05:43.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:42 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:43.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:42 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:43.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:42 smithi038 ceph-mon[26374]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:43.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:42 smithi038 ceph-mon[26374]: from='osd.4 [v2:172.21.15.98:6808/3945015359,v1:172.21.15.98:6809/3945015359]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T21:05:43.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:42 smithi038 ceph-mon[26374]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T21:05:44.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:44 smithi038 ceph-mon[26374]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T21:05:44.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:44 smithi038 ceph-mon[26374]: osdmap e33: 5 total, 4 up, 5 in 2024-04-22T21:05:44.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:44 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:05:44.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:44 smithi038 ceph-mon[26374]: from='osd.4 [v2:172.21.15.98:6808/3945015359,v1:172.21.15.98:6809/3945015359]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-22T21:05:44.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:44 smithi038 ceph-mon[26374]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-22T21:05:44.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:44 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:44.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:44 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:44.379 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 22 21:05:44 smithi098 ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-osd-4[46146]: 2024-04-22T21:05:44.004+0000 7fc6424ad700 -1 osd.4 0 waiting for initial osdmap 2024-04-22T21:05:44.379 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 22 21:05:44 smithi098 ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-osd-4[46146]: 2024-04-22T21:05:44.015+0000 7fc63d2a8700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:05:44.379 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:44 smithi098 ceph-mon[34053]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T21:05:44.380 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:44 smithi098 ceph-mon[34053]: osdmap e33: 5 total, 4 up, 5 in 2024-04-22T21:05:44.380 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:44 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:05:44.380 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:44 smithi098 ceph-mon[34053]: from='osd.4 [v2:172.21.15.98:6808/3945015359,v1:172.21.15.98:6809/3945015359]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-22T21:05:44.380 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:44 smithi098 ceph-mon[34053]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-22T21:05:44.380 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:44 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:44.380 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:44 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:45.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:45 smithi098 ceph-mon[34053]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-22T21:05:45.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:45 smithi098 ceph-mon[34053]: osdmap e34: 5 total, 4 up, 5 in 2024-04-22T21:05:45.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:45 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:05:45.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:45 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:05:45.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:45 smithi098 ceph-mon[34053]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:45.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:45 smithi098 ceph-mon[34053]: osd.4 [v2:172.21.15.98:6808/3945015359,v1:172.21.15.98:6809/3945015359] boot 2024-04-22T21:05:45.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:45 smithi098 ceph-mon[34053]: osdmap e35: 5 total, 5 up, 5 in 2024-04-22T21:05:45.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:45 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:05:45.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:45 smithi038 ceph-mon[26374]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-22T21:05:45.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:45 smithi038 ceph-mon[26374]: osdmap e34: 5 total, 4 up, 5 in 2024-04-22T21:05:45.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:05:45.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:05:45.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:45 smithi038 ceph-mon[26374]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-22T21:05:45.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:45 smithi038 ceph-mon[26374]: osd.4 [v2:172.21.15.98:6808/3945015359,v1:172.21.15.98:6809/3945015359] boot 2024-04-22T21:05:45.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:45 smithi038 ceph-mon[26374]: osdmap e35: 5 total, 5 up, 5 in 2024-04-22T21:05:45.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:05:46.153 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:46 smithi098 ceph-mon[34053]: purged_snaps scrub starts 2024-04-22T21:05:46.153 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:46 smithi098 ceph-mon[34053]: purged_snaps scrub ok 2024-04-22T21:05:46.154 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T21:05:46.154 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T21:05:46.154 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-22T21:05:46.154 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-22T21:05:46.154 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122993 s, 853 MB/s 2024-04-22T21:05:46.154 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-22T21:05:46.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:46 smithi038 ceph-mon[26374]: purged_snaps scrub starts 2024-04-22T21:05:46.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:46 smithi038 ceph-mon[26374]: purged_snaps scrub ok 2024-04-22T21:05:46.927 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2024-04-22T21:05:47.230 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:47 smithi098 ceph-mon[34053]: osdmap e36: 5 total, 5 up, 5 in 2024-04-22T21:05:47.231 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:47 smithi098 ceph-mon[34053]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:05:47.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:47 smithi038 ceph-mon[26374]: osdmap e36: 5 total, 5 up, 5 in 2024-04-22T21:05:47.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:47 smithi038 ceph-mon[26374]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:05:47.617 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi098/config 2024-04-22T21:05:48.816 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:48 smithi098 ceph-mon[34053]: Detected new or changed devices on smithi098 2024-04-22T21:05:48.817 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:48.817 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:48.817 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:05:48.817 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:05:48.817 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:48 smithi098 ceph-mon[34053]: Adjusting osd_memory_target on smithi098 to 7968M 2024-04-22T21:05:48.817 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:48.817 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:48.817 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:05:48.817 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:48.817 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:05:48.818 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:48.818 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:05:48.818 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:48.818 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:48 smithi098 ceph-mon[34053]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:05:48.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:48 smithi038 ceph-mon[26374]: Detected new or changed devices on smithi098 2024-04-22T21:05:48.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:48.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:48.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:05:48.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:05:48.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:48 smithi038 ceph-mon[26374]: Adjusting osd_memory_target on smithi098 to 7968M 2024-04-22T21:05:48.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:48.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:48.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:05:48.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:48.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:05:48.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:48.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:05:48.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:05:48.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:48 smithi038 ceph-mon[26374]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:05:49.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.113+0000 7f26c2aa1700 1 -- 172.21.15.98:0/2293040959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26bc0fbd00 msgr2=0x7f26bc0fc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:05:49.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.113+0000 7f26c2aa1700 1 --2- 172.21.15.98:0/2293040959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26bc0fbd00 0x7f26bc0fc110 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f26b0004940 tx=0x7f26b0009e30 comp rx=0 tx=0).stop 2024-04-22T21:05:49.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.113+0000 7f26c2aa1700 1 -- 172.21.15.98:0/2293040959 shutdown_connections 2024-04-22T21:05:49.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.113+0000 7f26c2aa1700 1 --2- 172.21.15.98:0/2293040959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26bc0fcfb0 0x7f26bc0fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:05:49.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.113+0000 7f26c2aa1700 1 --2- 172.21.15.98:0/2293040959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26bc0fbd00 0x7f26bc0fc110 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:05:49.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.113+0000 7f26c2aa1700 1 -- 172.21.15.98:0/2293040959 >> 172.21.15.98:0/2293040959 conn(0x7f26bc0f7480 msgr2=0x7f26bc0f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:05:49.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.113+0000 7f26c2aa1700 1 -- 172.21.15.98:0/2293040959 shutdown_connections 2024-04-22T21:05:49.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.113+0000 7f26c2aa1700 1 -- 172.21.15.98:0/2293040959 wait complete. 2024-04-22T21:05:49.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.113+0000 7f26c2aa1700 1 Processor -- start 2024-04-22T21:05:49.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.113+0000 7f26c2aa1700 1 -- start start 2024-04-22T21:05:49.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26c2aa1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26bc0fbd00 0x7f26bc066d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:05:49.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26c2aa1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26bc0fcfb0 0x7f26bc112770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:05:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26c2aa1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26bc112cb0 con 0x7f26bc0fcfb0 2024-04-22T21:05:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26c2aa1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26bc112df0 con 0x7f26bc0fbd00 2024-04-22T21:05:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26c083d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26bc0fbd00 0x7f26bc066d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:05:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26c083d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26bc0fbd00 0x7f26bc066d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51060/0 (socket says 172.21.15.98:51060) 2024-04-22T21:05:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26c083d700 1 -- 172.21.15.98:0/2309725512 learned_addr learned my addr 172.21.15.98:0/2309725512 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:05:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26bbfff700 1 --2- 172.21.15.98:0/2309725512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26bc0fcfb0 0x7f26bc112770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:05:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26c083d700 1 -- 172.21.15.98:0/2309725512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26bc0fcfb0 msgr2=0x7f26bc112770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:05:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26c083d700 1 --2- 172.21.15.98:0/2309725512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26bc0fcfb0 0x7f26bc112770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:05:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26c083d700 1 -- 172.21.15.98:0/2309725512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b0005040 con 0x7f26bc0fbd00 2024-04-22T21:05:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26c083d700 1 --2- 172.21.15.98:0/2309725512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26bc0fbd00 0x7f26bc066d20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f26b000b3b0 tx=0x7f26b0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:05:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26b97fa700 1 -- 172.21.15.98:0/2309725512 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26b000e040 con 0x7f26bc0fbd00 2024-04-22T21:05:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26c2aa1700 1 -- 172.21.15.98:0/2309725512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26bc112ff0 con 0x7f26bc0fbd00 2024-04-22T21:05:49.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26b97fa700 1 -- 172.21.15.98:0/2309725512 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26b000bcc0 con 0x7f26bc0fbd00 2024-04-22T21:05:49.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26b97fa700 1 -- 172.21.15.98:0/2309725512 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26b001c630 con 0x7f26bc0fbd00 2024-04-22T21:05:49.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.114+0000 7f26c2aa1700 1 -- 172.21.15.98:0/2309725512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26bc113520 con 0x7f26bc0fbd00 2024-04-22T21:05:49.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.115+0000 7f26c2aa1700 1 -- 172.21.15.98:0/2309725512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26bc04c710 con 0x7f26bc0fbd00 2024-04-22T21:05:49.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.116+0000 7f26b97fa700 1 -- 172.21.15.98:0/2309725512 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f26b0013980 con 0x7f26bc0fbd00 2024-04-22T21:05:49.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.116+0000 7f26b97fa700 1 --2- 172.21.15.98:0/2309725512 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f26ac0692a0 0x7f26ac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:05:49.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.116+0000 7f26b97fa700 1 -- 172.21.15.98:0/2309725512 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f26b0087ac0 con 0x7f26bc0fbd00 2024-04-22T21:05:49.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.116+0000 7f26bbfff700 1 --2- 172.21.15.98:0/2309725512 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f26ac0692a0 0x7f26ac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:05:49.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.117+0000 7f26bbfff700 1 --2- 172.21.15.98:0/2309725512 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f26ac0692a0 0x7f26ac06b750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f26a8009fd0 tx=0x7f26a8009f20 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:05:49.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.120+0000 7f26b97fa700 1 -- 172.21.15.98:0/2309725512 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f26b0057cf0 con 0x7f26bc0fbd00 2024-04-22T21:05:49.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:05:49.312+0000 7f26c2aa1700 1 -- 172.21.15.98:0/2309725512 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f26bc101320 con 0x7f26ac0692a0 2024-04-22T21:05:49.961 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:49 smithi098 ceph-mon[34053]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:05:49.961 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:49 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:05:49.961 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:49 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:05:49.962 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:49 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:50.124 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:49 smithi038 ceph-mon[26374]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:05:50.124 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:49 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:05:50.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:49 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:05:50.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:49 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:50.999 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:50 smithi098 ceph-mon[34053]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:05:51.124 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:50 smithi038 ceph-mon[26374]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:05:51.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:51 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/1254035417' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58e5fcf4-d0cc-4498-ae06-d51d8f2d0933"}]: dispatch 2024-04-22T21:05:51.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:51 smithi038 ceph-mon[26374]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58e5fcf4-d0cc-4498-ae06-d51d8f2d0933"}]: dispatch 2024-04-22T21:05:51.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:51 smithi038 ceph-mon[26374]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58e5fcf4-d0cc-4498-ae06-d51d8f2d0933"}]': finished 2024-04-22T21:05:51.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:51 smithi038 ceph-mon[26374]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T21:05:51.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:51 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:05:52.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:51 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/1254035417' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58e5fcf4-d0cc-4498-ae06-d51d8f2d0933"}]: dispatch 2024-04-22T21:05:52.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:51 smithi098 ceph-mon[34053]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58e5fcf4-d0cc-4498-ae06-d51d8f2d0933"}]: dispatch 2024-04-22T21:05:52.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:51 smithi098 ceph-mon[34053]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58e5fcf4-d0cc-4498-ae06-d51d8f2d0933"}]': finished 2024-04-22T21:05:52.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:51 smithi098 ceph-mon[34053]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T21:05:52.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:51 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:05:53.124 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:52 smithi038 ceph-mon[26374]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:05:53.124 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:52 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/3668206034' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:05:53.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:52 smithi098 ceph-mon[34053]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:05:53.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:52 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/3668206034' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:05:54.124 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:05:54.249 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:05:55.124 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:54 smithi038 ceph-mon[26374]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:05:55.249 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:54 smithi098 ceph-mon[34053]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:05:57.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:57 smithi038 ceph-mon[26374]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:05:57.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:57 smithi098 ceph-mon[34053]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:05:59.227 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:59 smithi098 ceph-mon[34053]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:05:59.227 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:59 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T21:05:59.228 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:05:59 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:05:59.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:59 smithi038 ceph-mon[26374]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:05:59.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:59 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T21:05:59.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:05:59 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:06:00.402 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:00 smithi098 ceph-mon[34053]: Deploying daemon osd.5 on smithi098 2024-04-22T21:06:00.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:00 smithi038 ceph-mon[26374]: Deploying daemon osd.5 on smithi098 2024-04-22T21:06:01.442 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:01 smithi098 ceph-mon[34053]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:06:01.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:01 smithi038 ceph-mon[26374]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:06:03.397 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:03 smithi098 ceph-mon[34053]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:06:03.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:03 smithi038 ceph-mon[26374]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:06:05.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:05 smithi098 ceph-mon[34053]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:06:05.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:05 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:05.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:05 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:05.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:05 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:06:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:05 smithi038 ceph-mon[26374]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:06:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:05 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:05 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:05 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:06:07.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:07.030+0000 7f26b97fa700 1 -- 172.21.15.98:0/2309725512 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f26bc101320 con 0x7f26ac0692a0 2024-04-22T21:06:07.026 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 5 on host 'smithi098' 2024-04-22T21:06:07.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:07.032+0000 7f26c2aa1700 1 -- 172.21.15.98:0/2309725512 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f26ac0692a0 msgr2=0x7f26ac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:07.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:07.032+0000 7f26c2aa1700 1 --2- 172.21.15.98:0/2309725512 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f26ac0692a0 0x7f26ac06b750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f26a8009fd0 tx=0x7f26a8009f20 comp rx=0 tx=0).stop 2024-04-22T21:06:07.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:07.032+0000 7f26c2aa1700 1 -- 172.21.15.98:0/2309725512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26bc0fbd00 msgr2=0x7f26bc066d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:07.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:07.032+0000 7f26c2aa1700 1 --2- 172.21.15.98:0/2309725512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26bc0fbd00 0x7f26bc066d20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f26b000b3b0 tx=0x7f26b0009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:07.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:07.032+0000 7f26c2aa1700 1 -- 172.21.15.98:0/2309725512 shutdown_connections 2024-04-22T21:06:07.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:07.032+0000 7f26c2aa1700 1 --2- 172.21.15.98:0/2309725512 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f26ac0692a0 0x7f26ac06b750 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:07.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:07.032+0000 7f26c2aa1700 1 --2- 172.21.15.98:0/2309725512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26bc0fbd00 0x7f26bc066d20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:07.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:07.032+0000 7f26c2aa1700 1 --2- 172.21.15.98:0/2309725512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26bc0fcfb0 0x7f26bc112770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:07.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:07.032+0000 7f26c2aa1700 1 -- 172.21.15.98:0/2309725512 >> 172.21.15.98:0/2309725512 conn(0x7f26bc0f7480 msgr2=0x7f26bc0f8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:07.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:07.032+0000 7f26c2aa1700 1 -- 172.21.15.98:0/2309725512 shutdown_connections 2024-04-22T21:06:07.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:07.032+0000 7f26c2aa1700 1 -- 172.21.15.98:0/2309725512 wait complete. 2024-04-22T21:06:07.249 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:07 smithi098 ceph-mon[34053]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:06:07.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:07 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:07.250 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:07 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:07.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:07 smithi038 ceph-mon[26374]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:06:07.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:07 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:07.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:07 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:07.677 DEBUG:teuthology.orchestra.run.smithi098:osd.5> sudo journalctl -f -n 0 -u ceph-3c5f5352-00eb-11ef-bc93-c7b262605968@osd.5.service 2024-04-22T21:06:07.679 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-22T21:06:07.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T21:06:07.957 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:08.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:08 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:06:08.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:08 smithi038 ceph-mon[26374]: from='osd.5 [v2:172.21.15.98:6816/2442541230,v1:172.21.15.98:6817/2442541230]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T21:06:08.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:08 smithi038 ceph-mon[26374]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T21:06:08.493 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:08 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:06:08.493 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:08 smithi098 ceph-mon[34053]: from='osd.5 [v2:172.21.15.98:6816/2442541230,v1:172.21.15.98:6817/2442541230]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T21:06:08.494 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:08 smithi098 ceph-mon[34053]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T21:06:08.494 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 22 21:06:08 smithi098 ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-osd-5[52624]: 2024-04-22T21:06:08.107+0000 7f4464c2a540 -1 osd.5 0 log_to_monitors true 2024-04-22T21:06:09.369 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:09 smithi038 ceph-mon[26374]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:06:09.369 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:09 smithi038 ceph-mon[26374]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T21:06:09.369 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:09 smithi038 ceph-mon[26374]: osdmap e38: 6 total, 5 up, 6 in 2024-04-22T21:06:09.369 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:09 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:06:09.369 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:09 smithi038 ceph-mon[26374]: from='osd.5 [v2:172.21.15.98:6816/2442541230,v1:172.21.15.98:6817/2442541230]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-22T21:06:09.369 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:09 smithi038 ceph-mon[26374]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-22T21:06:09.369 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:09 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:09.370 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:09 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:09.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:09 smithi098 ceph-mon[34053]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:06:09.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:09 smithi098 ceph-mon[34053]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T21:06:09.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:09 smithi098 ceph-mon[34053]: osdmap e38: 6 total, 5 up, 6 in 2024-04-22T21:06:09.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:09 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:06:09.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:09 smithi098 ceph-mon[34053]: from='osd.5 [v2:172.21.15.98:6816/2442541230,v1:172.21.15.98:6817/2442541230]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-22T21:06:09.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:09 smithi098 ceph-mon[34053]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-22T21:06:09.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:09 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:09.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:09 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:09.501 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 22 21:06:09 smithi098 ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-osd-5[52624]: 2024-04-22T21:06:09.231+0000 7f445b3fc700 -1 osd.5 0 waiting for initial osdmap 2024-04-22T21:06:09.501 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 22 21:06:09 smithi098 ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-osd-5[52624]: 2024-04-22T21:06:09.238+0000 7f44541f3700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:06:09.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.686+0000 7fd3b3583700 1 -- 172.21.15.38:0/92190497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac0fbd40 msgr2=0x7fd3ac0fc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:09.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.686+0000 7fd3b3583700 1 --2- 172.21.15.38:0/92190497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac0fbd40 0x7fd3ac0fc150 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009ce0 tx=0x7fd3a4004d10 comp rx=0 tx=0).stop 2024-04-22T21:06:09.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.686+0000 7fd3b3583700 1 -- 172.21.15.38:0/92190497 shutdown_connections 2024-04-22T21:06:09.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.686+0000 7fd3b3583700 1 --2- 172.21.15.38:0/92190497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3ac0fcff0 0x7fd3ac0fd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:09.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.686+0000 7fd3b3583700 1 --2- 172.21.15.38:0/92190497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac0fbd40 0x7fd3ac0fc150 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:09.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.686+0000 7fd3b3583700 1 -- 172.21.15.38:0/92190497 >> 172.21.15.38:0/92190497 conn(0x7fd3ac0f74c0 msgr2=0x7fd3ac0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:09.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.686+0000 7fd3b3583700 1 -- 172.21.15.38:0/92190497 shutdown_connections 2024-04-22T21:06:09.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.686+0000 7fd3b3583700 1 -- 172.21.15.38:0/92190497 wait complete. 2024-04-22T21:06:09.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.687+0000 7fd3b3583700 1 Processor -- start 2024-04-22T21:06:09.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.687+0000 7fd3b3583700 1 -- start start 2024-04-22T21:06:09.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.687+0000 7fd3b3583700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac0fbd40 0x7fd3ac110760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:09.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.687+0000 7fd3b3583700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3ac0fcff0 0x7fd3ac112cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:09.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.687+0000 7fd3b3583700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac066360 con 0x7fd3ac0fbd40 2024-04-22T21:06:09.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.687+0000 7fd3b3583700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac0664a0 con 0x7fd3ac0fcff0 2024-04-22T21:06:09.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.687+0000 7fd3b131f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac0fbd40 0x7fd3ac110760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:09.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.687+0000 7fd3b131f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac0fbd40 0x7fd3ac110760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49600/0 (socket says 172.21.15.38:49600) 2024-04-22T21:06:09.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.687+0000 7fd3b131f700 1 -- 172.21.15.38:0/3554887126 learned_addr learned my addr 172.21.15.38:0/3554887126 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:09.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.687+0000 7fd3b0b1e700 1 --2- 172.21.15.38:0/3554887126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3ac0fcff0 0x7fd3ac112cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:09.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.687+0000 7fd3b131f700 1 -- 172.21.15.38:0/3554887126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3ac0fcff0 msgr2=0x7fd3ac112cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:09.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.687+0000 7fd3b131f700 1 --2- 172.21.15.38:0/3554887126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3ac0fcff0 0x7fd3ac112cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:09.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.687+0000 7fd3b131f700 1 -- 172.21.15.38:0/3554887126 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a4009940 con 0x7fd3ac0fbd40 2024-04-22T21:06:09.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.688+0000 7fd3b131f700 1 --2- 172.21.15.38:0/3554887126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac0fbd40 0x7fd3ac110760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400bfd0 tx=0x7fd3a4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:09.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.688+0000 7fd3a9ffb700 1 -- 172.21.15.38:0/3554887126 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3a400e040 con 0x7fd3ac0fbd40 2024-04-22T21:06:09.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.688+0000 7fd3a9ffb700 1 -- 172.21.15.38:0/3554887126 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3a401cd00 con 0x7fd3ac0fbd40 2024-04-22T21:06:09.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.688+0000 7fd3b3583700 1 -- 172.21.15.38:0/3554887126 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ac1131f0 con 0x7fd3ac0fbd40 2024-04-22T21:06:09.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.688+0000 7fd3a9ffb700 1 -- 172.21.15.38:0/3554887126 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3a4012c40 con 0x7fd3ac0fbd40 2024-04-22T21:06:09.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.688+0000 7fd3b3583700 1 -- 172.21.15.38:0/3554887126 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ac1137b0 con 0x7fd3ac0fbd40 2024-04-22T21:06:09.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.689+0000 7fd3b3583700 1 -- 172.21.15.38:0/3554887126 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ac05f5f0 con 0x7fd3ac0fbd40 2024-04-22T21:06:09.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.690+0000 7fd3a9ffb700 1 -- 172.21.15.38:0/3554887126 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fd3a40124a0 con 0x7fd3ac0fbd40 2024-04-22T21:06:09.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.690+0000 7fd3a9ffb700 1 --2- 172.21.15.38:0/3554887126 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fd3940692a0 0x7fd39406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:09.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.690+0000 7fd3a9ffb700 1 -- 172.21.15.38:0/3554887126 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4354+0+0 (secure 0 0 0) 0x7fd3a4088ff0 con 0x7fd3ac0fbd40 2024-04-22T21:06:09.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.690+0000 7fd3b0b1e700 1 --2- 172.21.15.38:0/3554887126 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fd3940692a0 0x7fd39406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:09.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.690+0000 7fd3b0b1e700 1 --2- 172.21.15.38:0/3554887126 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fd3940692a0 0x7fd39406b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009b80 tx=0x7fd3a000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:09.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.695+0000 7fd3a9ffb700 1 -- 172.21.15.38:0/3554887126 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd3a4055710 con 0x7fd3ac0fbd40 2024-04-22T21:06:09.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.891+0000 7fd3b3583700 1 -- 172.21.15.38:0/3554887126 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fd3ac04bed0 con 0x7fd3ac0fbd40 2024-04-22T21:06:09.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.891+0000 7fd3a9ffb700 1 -- 172.21.15.38:0/3554887126 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7fd3a4058d30 con 0x7fd3ac0fbd40 2024-04-22T21:06:09.888 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:06:09.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.894+0000 7fd3b3583700 1 -- 172.21.15.38:0/3554887126 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fd3940692a0 msgr2=0x7fd39406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:09.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.894+0000 7fd3b3583700 1 --2- 172.21.15.38:0/3554887126 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fd3940692a0 0x7fd39406b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009b80 tx=0x7fd3a000b040 comp rx=0 tx=0).stop 2024-04-22T21:06:09.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.894+0000 7fd3b3583700 1 -- 172.21.15.38:0/3554887126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac0fbd40 msgr2=0x7fd3ac110760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:09.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.894+0000 7fd3b3583700 1 --2- 172.21.15.38:0/3554887126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac0fbd40 0x7fd3ac110760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400bfd0 tx=0x7fd3a4004d10 comp rx=0 tx=0).stop 2024-04-22T21:06:09.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.894+0000 7fd3b3583700 1 -- 172.21.15.38:0/3554887126 shutdown_connections 2024-04-22T21:06:09.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.894+0000 7fd3b3583700 1 --2- 172.21.15.38:0/3554887126 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fd3940692a0 0x7fd39406b750 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:09.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.894+0000 7fd3b3583700 1 --2- 172.21.15.38:0/3554887126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac0fbd40 0x7fd3ac110760 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:09.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.894+0000 7fd3b3583700 1 --2- 172.21.15.38:0/3554887126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3ac0fcff0 0x7fd3ac112cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:09.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.894+0000 7fd3b3583700 1 -- 172.21.15.38:0/3554887126 >> 172.21.15.38:0/3554887126 conn(0x7fd3ac0f74c0 msgr2=0x7fd3ac0f8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:09.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.894+0000 7fd3b3583700 1 -- 172.21.15.38:0/3554887126 shutdown_connections 2024-04-22T21:06:09.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:09.894+0000 7fd3b3583700 1 -- 172.21.15.38:0/3554887126 wait complete. 2024-04-22T21:06:10.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:10 smithi098 ceph-mon[34053]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-22T21:06:10.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:10 smithi098 ceph-mon[34053]: osdmap e39: 6 total, 5 up, 6 in 2024-04-22T21:06:10.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:10 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:06:10.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:10 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:06:10.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:10 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/3554887126' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T21:06:10.568 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:10 smithi038 ceph-mon[26374]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-22T21:06:10.568 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:10 smithi038 ceph-mon[26374]: osdmap e39: 6 total, 5 up, 6 in 2024-04-22T21:06:10.568 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:10 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:06:10.568 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:10 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:06:10.568 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:10 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3554887126' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T21:06:10.595 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":39,"num_osds":6,"num_up_osds":5,"osd_up_since":1713819944,"num_in_osds":6,"osd_in_since":1713819951,"num_remapped_pgs":0} 2024-04-22T21:06:11.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:11 smithi098 ceph-mon[34053]: purged_snaps scrub starts 2024-04-22T21:06:11.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:11 smithi098 ceph-mon[34053]: purged_snaps scrub ok 2024-04-22T21:06:11.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:11 smithi098 ceph-mon[34053]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:06:11.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:11 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:06:11.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:11 smithi098 ceph-mon[34053]: osd.5 [v2:172.21.15.98:6816/2442541230,v1:172.21.15.98:6817/2442541230] boot 2024-04-22T21:06:11.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:11 smithi098 ceph-mon[34053]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T21:06:11.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:11 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:06:11.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:11 smithi038 ceph-mon[26374]: purged_snaps scrub starts 2024-04-22T21:06:11.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:11 smithi038 ceph-mon[26374]: purged_snaps scrub ok 2024-04-22T21:06:11.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:11 smithi038 ceph-mon[26374]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-22T21:06:11.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:11 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:06:11.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:11 smithi038 ceph-mon[26374]: osd.5 [v2:172.21.15.98:6816/2442541230,v1:172.21.15.98:6817/2442541230] boot 2024-04-22T21:06:11.537 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:11 smithi038 ceph-mon[26374]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T21:06:11.537 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:11 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:06:11.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T21:06:11.874 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:12.475 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:12 smithi098 ceph-mon[34053]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T21:06:12.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:12 smithi038 ceph-mon[26374]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T21:06:13.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.525+0000 7f7e366ca700 1 -- 172.21.15.38:0/348699996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e300fcf20 msgr2=0x7f7e300fd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:13.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.525+0000 7f7e366ca700 1 --2- 172.21.15.38:0/348699996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e300fcf20 0x7f7e300fd330 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7e20004940 tx=0x7f7e20009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:13.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.525+0000 7f7e366ca700 1 -- 172.21.15.38:0/348699996 shutdown_connections 2024-04-22T21:06:13.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.525+0000 7f7e366ca700 1 --2- 172.21.15.38:0/348699996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e300fe090 0x7f7e300fe500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:13.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.525+0000 7f7e366ca700 1 --2- 172.21.15.38:0/348699996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e300fcf20 0x7f7e300fd330 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:13.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.525+0000 7f7e366ca700 1 -- 172.21.15.38:0/348699996 >> 172.21.15.38:0/348699996 conn(0x7f7e300f85c0 msgr2=0x7f7e300faa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:13.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.525+0000 7f7e366ca700 1 -- 172.21.15.38:0/348699996 shutdown_connections 2024-04-22T21:06:13.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.525+0000 7f7e366ca700 1 -- 172.21.15.38:0/348699996 wait complete. 2024-04-22T21:06:13.522 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:13 smithi038 ceph-mon[26374]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:13.523 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:13.523 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:13.523 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:06:13.523 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:06:13.523 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:06:13.523 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:13.523 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:06:13.523 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:06:13.523 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:13.523 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:06:13.523 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:06:13.523 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:06:13.523 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:13 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:13.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.526+0000 7f7e366ca700 1 Processor -- start 2024-04-22T21:06:13.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.526+0000 7f7e366ca700 1 -- start start 2024-04-22T21:06:13.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.526+0000 7f7e366ca700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e300fcf20 0x7f7e3010d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:13.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.526+0000 7f7e366ca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e300fe090 0x7f7e3010df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:13.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.526+0000 7f7e366ca700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3010e520 con 0x7f7e300fcf20 2024-04-22T21:06:13.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.526+0000 7f7e366ca700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3010ed70 con 0x7f7e300fe090 2024-04-22T21:06:13.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.526+0000 7f7e2ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e300fcf20 0x7f7e3010d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:13.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.526+0000 7f7e2ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e300fcf20 0x7f7e3010d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58582/0 (socket says 172.21.15.38:58582) 2024-04-22T21:06:13.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.526+0000 7f7e2ffff700 1 -- 172.21.15.38:0/2204475511 learned_addr learned my addr 172.21.15.38:0/2204475511 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:13.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.526+0000 7f7e2f7fe700 1 --2- 172.21.15.38:0/2204475511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e300fe090 0x7f7e3010df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:13.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.527+0000 7f7e2ffff700 1 -- 172.21.15.38:0/2204475511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e300fe090 msgr2=0x7f7e3010df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:13.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.527+0000 7f7e2ffff700 1 --2- 172.21.15.38:0/2204475511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e300fe090 0x7f7e3010df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:13.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.527+0000 7f7e2ffff700 1 -- 172.21.15.38:0/2204475511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e20005040 con 0x7f7e300fcf20 2024-04-22T21:06:13.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.527+0000 7f7e2f7fe700 1 --2- 172.21.15.38:0/2204475511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e300fe090 0x7f7e3010df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T21:06:13.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.527+0000 7f7e2ffff700 1 --2- 172.21.15.38:0/2204475511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e300fcf20 0x7f7e3010d9e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7e2000b3b0 tx=0x7f7e20009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:13.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.527+0000 7f7e2cff9700 1 -- 172.21.15.38:0/2204475511 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e2000e040 con 0x7f7e300fcf20 2024-04-22T21:06:13.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.527+0000 7f7e2cff9700 1 -- 172.21.15.38:0/2204475511 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e20004290 con 0x7f7e300fcf20 2024-04-22T21:06:13.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.527+0000 7f7e2cff9700 1 -- 172.21.15.38:0/2204475511 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e2001c3f0 con 0x7f7e300fcf20 2024-04-22T21:06:13.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.527+0000 7f7e366ca700 1 -- 172.21.15.38:0/2204475511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e3010eff0 con 0x7f7e300fcf20 2024-04-22T21:06:13.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.527+0000 7f7e366ca700 1 -- 172.21.15.38:0/2204475511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3010f530 con 0x7f7e300fcf20 2024-04-22T21:06:13.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.528+0000 7f7e366ca700 1 -- 172.21.15.38:0/2204475511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3005f5f0 con 0x7f7e300fcf20 2024-04-22T21:06:13.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.529+0000 7f7e2cff9700 1 -- 172.21.15.38:0/2204475511 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f7e2000bbf0 con 0x7f7e300fcf20 2024-04-22T21:06:13.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.529+0000 7f7e2cff9700 1 --2- 172.21.15.38:0/2204475511 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7e180692b0 0x7f7e1806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:13.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.529+0000 7f7e2cff9700 1 -- 172.21.15.38:0/2204475511 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7e200880c0 con 0x7f7e300fcf20 2024-04-22T21:06:13.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.529+0000 7f7e2f7fe700 1 --2- 172.21.15.38:0/2204475511 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7e180692b0 0x7f7e1806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:13.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.530+0000 7f7e2f7fe700 1 --2- 172.21.15.38:0/2204475511 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7e180692b0 0x7f7e1806b760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009fd0 tx=0x7f7e2400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:13.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.534+0000 7f7e2cff9700 1 -- 172.21.15.38:0/2204475511 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f7e20058260 con 0x7f7e300fcf20 2024-04-22T21:06:13.642 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:13 smithi098 ceph-mon[34053]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:13.642 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:13.642 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:13.642 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:06:13.643 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:06:13.643 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:06:13.643 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:13.643 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:06:13.643 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:06:13.643 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:13.643 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:06:13.643 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:06:13.643 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:06:13.643 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:13 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:06:13.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.728+0000 7f7e366ca700 1 -- 172.21.15.38:0/2204475511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f7e30105910 con 0x7f7e300fcf20 2024-04-22T21:06:13.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.729+0000 7f7e2cff9700 1 -- 172.21.15.38:0/2204475511 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v41) v1 ==== 74+0+130 (secure 0 0 0) 0x7f7e20057df0 con 0x7f7e300fcf20 2024-04-22T21:06:13.725 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:06:13.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.731+0000 7f7e366ca700 1 -- 172.21.15.38:0/2204475511 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7e180692b0 msgr2=0x7f7e1806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:13.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.731+0000 7f7e366ca700 1 --2- 172.21.15.38:0/2204475511 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7e180692b0 0x7f7e1806b760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009fd0 tx=0x7f7e2400b040 comp rx=0 tx=0).stop 2024-04-22T21:06:13.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.731+0000 7f7e366ca700 1 -- 172.21.15.38:0/2204475511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e300fcf20 msgr2=0x7f7e3010d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:13.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.731+0000 7f7e366ca700 1 --2- 172.21.15.38:0/2204475511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e300fcf20 0x7f7e3010d9e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7e2000b3b0 tx=0x7f7e20009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:13.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.731+0000 7f7e366ca700 1 -- 172.21.15.38:0/2204475511 shutdown_connections 2024-04-22T21:06:13.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.731+0000 7f7e366ca700 1 --2- 172.21.15.38:0/2204475511 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7e180692b0 0x7f7e1806b760 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:13.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.731+0000 7f7e366ca700 1 --2- 172.21.15.38:0/2204475511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e300fcf20 0x7f7e3010d9e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:13.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.731+0000 7f7e366ca700 1 --2- 172.21.15.38:0/2204475511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e300fe090 0x7f7e3010df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:13.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.731+0000 7f7e366ca700 1 -- 172.21.15.38:0/2204475511 >> 172.21.15.38:0/2204475511 conn(0x7f7e300f85c0 msgr2=0x7f7e300f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:13.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.731+0000 7f7e366ca700 1 -- 172.21.15.38:0/2204475511 shutdown_connections 2024-04-22T21:06:13.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:13.731+0000 7f7e366ca700 1 -- 172.21.15.38:0/2204475511 wait complete. 2024-04-22T21:06:14.332 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":41,"num_osds":6,"num_up_osds":6,"osd_up_since":1713819970,"num_in_osds":6,"osd_in_since":1713819951,"num_remapped_pgs":0} 2024-04-22T21:06:14.333 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T21:06:14.596 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:14.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:14 smithi038 ceph-mon[26374]: Detected new or changed devices on smithi098 2024-04-22T21:06:14.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:14 smithi038 ceph-mon[26374]: Adjusting osd_memory_target on smithi098 to 5312M 2024-04-22T21:06:14.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:14 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/2204475511' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T21:06:14.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:14 smithi098 ceph-mon[34053]: Detected new or changed devices on smithi098 2024-04-22T21:06:14.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:14 smithi098 ceph-mon[34053]: Adjusting osd_memory_target on smithi098 to 5312M 2024-04-22T21:06:14.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:14 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/2204475511' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T21:06:15.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:15 smithi038 ceph-mon[26374]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:15.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:15 smithi098 ceph-mon[34053]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:16.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.172+0000 7f216fb7c700 1 -- 172.21.15.38:0/2401798316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21680fb4d0 msgr2=0x7f21680fb920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:16.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.172+0000 7f216fb7c700 1 --2- 172.21.15.38:0/2401798316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21680fb4d0 0x7f21680fb920 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f215c004940 tx=0x7f215c009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:16.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.172+0000 7f216fb7c700 1 -- 172.21.15.38:0/2401798316 shutdown_connections 2024-04-22T21:06:16.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.172+0000 7f216fb7c700 1 --2- 172.21.15.38:0/2401798316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21680fb4d0 0x7f21680fb920 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:16.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.172+0000 7f216fb7c700 1 --2- 172.21.15.38:0/2401798316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2168102f60 0x7f2168103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:16.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.172+0000 7f216fb7c700 1 -- 172.21.15.38:0/2401798316 >> 172.21.15.38:0/2401798316 conn(0x7f21680714c0 msgr2=0x7f21680718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:16.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.172+0000 7f216fb7c700 1 -- 172.21.15.38:0/2401798316 shutdown_connections 2024-04-22T21:06:16.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.172+0000 7f216fb7c700 1 -- 172.21.15.38:0/2401798316 wait complete. 2024-04-22T21:06:16.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.172+0000 7f216fb7c700 1 Processor -- start 2024-04-22T21:06:16.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.172+0000 7f216fb7c700 1 -- start start 2024-04-22T21:06:16.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.173+0000 7f216fb7c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21680fb4d0 0x7f2168113960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:16.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.173+0000 7f216fb7c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2168102f60 0x7f2168113ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:16.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.173+0000 7f216fb7c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21681144f0 con 0x7f2168102f60 2024-04-22T21:06:16.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.173+0000 7f216fb7c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216810daf0 con 0x7f21680fb4d0 2024-04-22T21:06:16.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.173+0000 7f216d117700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2168102f60 0x7f2168113ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:16.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.173+0000 7f216d117700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2168102f60 0x7f2168113ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58596/0 (socket says 172.21.15.38:58596) 2024-04-22T21:06:16.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.173+0000 7f216d117700 1 -- 172.21.15.38:0/3092197812 learned_addr learned my addr 172.21.15.38:0/3092197812 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:16.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.173+0000 7f216d918700 1 --2- 172.21.15.38:0/3092197812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21680fb4d0 0x7f2168113960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:16.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.173+0000 7f216d117700 1 -- 172.21.15.38:0/3092197812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21680fb4d0 msgr2=0x7f2168113960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:16.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.173+0000 7f216d117700 1 --2- 172.21.15.38:0/3092197812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21680fb4d0 0x7f2168113960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:16.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.173+0000 7f216d117700 1 -- 172.21.15.38:0/3092197812 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215c005040 con 0x7f2168102f60 2024-04-22T21:06:16.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.173+0000 7f216d117700 1 --2- 172.21.15.38:0/3092197812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2168102f60 0x7f2168113ea0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f215c00b3b0 tx=0x7f215c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:16.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.173+0000 7f215a7fc700 1 -- 172.21.15.38:0/3092197812 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f215c00e040 con 0x7f2168102f60 2024-04-22T21:06:16.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.173+0000 7f215a7fc700 1 -- 172.21.15.38:0/3092197812 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f215c00bbc0 con 0x7f2168102f60 2024-04-22T21:06:16.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.173+0000 7f216fb7c700 1 -- 172.21.15.38:0/3092197812 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216810dd70 con 0x7f2168102f60 2024-04-22T21:06:16.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.173+0000 7f215a7fc700 1 -- 172.21.15.38:0/3092197812 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f215c01c3f0 con 0x7f2168102f60 2024-04-22T21:06:16.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.174+0000 7f216fb7c700 1 -- 172.21.15.38:0/3092197812 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216810e330 con 0x7f2168102f60 2024-04-22T21:06:16.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.174+0000 7f216fb7c700 1 -- 172.21.15.38:0/3092197812 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2168105340 con 0x7f2168102f60 2024-04-22T21:06:16.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.175+0000 7f215a7fc700 1 -- 172.21.15.38:0/3092197812 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f215c00bd30 con 0x7f2168102f60 2024-04-22T21:06:16.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.175+0000 7f215a7fc700 1 --2- 172.21.15.38:0/3092197812 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f21540692a0 0x7f215406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:16.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.175+0000 7f215a7fc700 1 -- 172.21.15.38:0/3092197812 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f215c088030 con 0x7f2168102f60 2024-04-22T21:06:16.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.175+0000 7f216d918700 1 --2- 172.21.15.38:0/3092197812 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f21540692a0 0x7f215406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:16.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.176+0000 7f216d918700 1 --2- 172.21.15.38:0/3092197812 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f21540692a0 0x7f215406b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2164007fd0 tx=0x7f2164009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:16.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.180+0000 7f215a7fc700 1 -- 172.21.15.38:0/3092197812 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f215c0581e0 con 0x7f2168102f60 2024-04-22T21:06:16.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.376+0000 7f216fb7c700 1 -- 172.21.15.38:0/3092197812 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f216810efb0 con 0x7f2168102f60 2024-04-22T21:06:16.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.377+0000 7f215a7fc700 1 -- 172.21.15.38:0/3092197812 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11022 (secure 0 0 0) 0x7f215c057d70 con 0x7f2168102f60 2024-04-22T21:06:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:06:16.373 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":41,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","created":"2024-04-22T21:00:25.280047+0000","modified":"2024-04-22T21:06:11.235267+0000","last_up_change":"2024-04-22T21:06:10.229585+0000","last_in_change":"2024-04-22T21:05:51.689258+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T21:04:56.557561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"aeb7d8ce-e081-4a6a-8c2e-560bcb819a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":454874338},{"type":"v1","addr":"172.21.15.38:6803","nonce":454874338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":454874338},{"type":"v1","addr":"172.21.15.38:6805","nonce":454874338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":454874338},{"type":"v1","addr":"172.21.15.38:6809","nonce":454874338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":454874338},{"type":"v1","addr":"172.21.15.38:6807","nonce":454874338}]},"public_addr":"172.21.15.38:6803/454874338","cluster_addr":"172.21.15.38:6805/454874338","heartbeat_back_addr":"172.21.15.38:6809/454874338","heartbeat_front_addr":"172.21.15.38:6807/454874338","state":["exists","up"]},{"osd":1,"uuid":"7c90956f-6cb0-4a71-86c0-9c38e9710808","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1336148940},{"type":"v1","addr":"172.21.15.38:6811","nonce":1336148940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1336148940},{"type":"v1","addr":"172.21.15.38:6813","nonce":1336148940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1336148940},{"type":"v1","addr":"172.21.15.38:6817","nonce":1336148940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1336148940},{"type":"v1","addr":"172.21.15.38:6815","nonce":1336148940}]},"public_addr":"172.21.15.38:6811/1336148940","cluster_addr":"172.21.15.38:6813/1336148940","heartbeat_back_addr":"172.21.15.38:6817/1336148940","heartbeat_front_addr":"172.21.15.38:6815/1336148940","state":["exists","up"]},{"osd":2,"uuid":"379616b9-bdc7-4fd6-a1c5-b937e2ef3202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":47489714},{"type":"v1","addr":"172.21.15.38:6819","nonce":47489714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":47489714},{"type":"v1","addr":"172.21.15.38:6821","nonce":47489714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":47489714},{"type":"v1","addr":"172.21.15.38:6825","nonce":47489714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":47489714},{"type":"v1","addr":"172.21.15.38:6823","nonce":47489714}]},"public_addr":"172.21.15.38:6819/47489714","cluster_addr":"172.21.15.38:6821/47489714","heartbeat_back_addr":"172.21.15.38:6825/47489714","heartbeat_front_addr":"172.21.15.38:6823/47489714","state":["exists","up"]},{"osd":3,"uuid":"becb1602-d471-4c38-8cd1-781828c4b2f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2538219727},{"type":"v1","addr":"172.21.15.98:6801","nonce":2538219727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2538219727},{"type":"v1","addr":"172.21.15.98:6803","nonce":2538219727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2538219727},{"type":"v1","addr":"172.21.15.98:6807","nonce":2538219727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2538219727},{"type":"v1","addr":"172.21.15.98:6805","nonce":2538219727}]},"public_addr":"172.21.15.98:6801/2538219727","cluster_addr":"172.21.15.98:6803/2538219727","heartbeat_back_addr":"172.21.15.98:6807/2538219727","heartbeat_front_addr":"172.21.15.98:6805/2538219727","state":["exists","up"]},{"osd":4,"uuid":"c886eb15-63f1-4bb9-a229-0278c8372f5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3945015359},{"type":"v1","addr":"172.21.15.98:6809","nonce":3945015359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3945015359},{"type":"v1","addr":"172.21.15.98:6811","nonce":3945015359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3945015359},{"type":"v1","addr":"172.21.15.98:6815","nonce":3945015359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3945015359},{"type":"v1","addr":"172.21.15.98:6813","nonce":3945015359}]},"public_addr":"172.21.15.98:6809/3945015359","cluster_addr":"172.21.15.98:6811/3945015359","heartbeat_back_addr":"172.21.15.98:6815/3945015359","heartbeat_front_addr":"172.21.15.98:6813/3945015359","state":["exists","up"]},{"osd":5,"uuid":"58e5fcf4-d0cc-4498-ae06-d51d8f2d0933","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2442541230},{"type":"v1","addr":"172.21.15.98:6817","nonce":2442541230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2442541230},{"type":"v1","addr":"172.21.15.98:6819","nonce":2442541230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2442541230},{"type":"v1","addr":"172.21.15.98:6823","nonce":2442541230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2442541230},{"type":"v1","addr":"172.21.15.98:6821","nonce":2442541230}]},"public_addr":"172.21.15.98:6817/2442541230","cluster_addr":"172.21.15.98:6819/2442541230","heartbeat_back_addr":"172.21.15.98:6823/2442541230","heartbeat_front_addr":"172.21.15.98:6821/2442541230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:03:53.821993+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:04:24.335225+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:04:52.351848+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:05:17.370839+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:05:43.310131+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:06:09.074139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/3596300465":"2024-04-23T21:02:38.036748+0000","172.21.15.38:6801/989134174":"2024-04-23T21:02:38.036748+0000","172.21.15.38:6800/989134174":"2024-04-23T21:02:38.036748+0000","172.21.15.38:0/420277838":"2024-04-23T21:02:38.036748+0000","172.21.15.38:0/3486303720":"2024-04-23T21:01:31.045112+0000","172.21.15.38:6800/2551561659":"2024-04-23T21:01:31.045112+0000","172.21.15.38:0/2835639508":"2024-04-23T21:01:31.045112+0000","172.21.15.38:6801/2787760150":"2024-04-23T21:00:59.025532+0000","172.21.15.38:0/3765130717":"2024-04-23T21:00:59.025532+0000","172.21.15.38:6801/2551561659":"2024-04-23T21:01:31.045112+0000","172.21.15.38:6800/2787760150":"2024-04-23T21:00:59.025532+0000","172.21.15.38:0/2942485563":"2024-04-23T21:00:59.025532+0000","172.21.15.38:0/4050800451":"2024-04-23T21:02:38.036748+0000","172.21.15.38:0/2754521346":"2024-04-23T21:01:31.045112+0000","172.21.15.38:0/3342218679":"2024-04-23T21:00:59.025532+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T21:06:16.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.379+0000 7f216fb7c700 1 -- 172.21.15.38:0/3092197812 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f21540692a0 msgr2=0x7f215406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:16.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.379+0000 7f216fb7c700 1 --2- 172.21.15.38:0/3092197812 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f21540692a0 0x7f215406b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2164007fd0 tx=0x7f2164009040 comp rx=0 tx=0).stop 2024-04-22T21:06:16.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.379+0000 7f216fb7c700 1 -- 172.21.15.38:0/3092197812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2168102f60 msgr2=0x7f2168113ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:16.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.379+0000 7f216fb7c700 1 --2- 172.21.15.38:0/3092197812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2168102f60 0x7f2168113ea0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f215c00b3b0 tx=0x7f215c009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:16.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.379+0000 7f216fb7c700 1 -- 172.21.15.38:0/3092197812 shutdown_connections 2024-04-22T21:06:16.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.379+0000 7f216fb7c700 1 --2- 172.21.15.38:0/3092197812 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f21540692a0 0x7f215406b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:16.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.379+0000 7f216fb7c700 1 --2- 172.21.15.38:0/3092197812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21680fb4d0 0x7f2168113960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:16.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.379+0000 7f216fb7c700 1 --2- 172.21.15.38:0/3092197812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2168102f60 0x7f2168113ea0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:16.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.379+0000 7f216fb7c700 1 -- 172.21.15.38:0/3092197812 >> 172.21.15.38:0/3092197812 conn(0x7f21680714c0 msgr2=0x7f21680fa7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:16.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.379+0000 7f216fb7c700 1 -- 172.21.15.38:0/3092197812 shutdown_connections 2024-04-22T21:06:16.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:16.379+0000 7f216fb7c700 1 -- 172.21.15.38:0/3092197812 wait complete. 2024-04-22T21:06:16.983 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T21:04:56.557561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-22T21:06:16.983 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-22T21:06:17.250 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:17.521 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:17 smithi038 ceph-mon[26374]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:17.521 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:17 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3092197812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:06:17.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:17 smithi098 ceph-mon[34053]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:17.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:17 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/3092197812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:06:18.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.822+0000 7ff89eb88700 1 -- 172.21.15.38:0/1600336752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8980fde70 msgr2=0x7ff8980fe240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:18.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.822+0000 7ff89eb88700 1 --2- 172.21.15.38:0/1600336752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8980fde70 0x7ff8980fe240 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff890009ce0 tx=0x7ff890004d10 comp rx=0 tx=0).stop 2024-04-22T21:06:18.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.822+0000 7ff89eb88700 1 -- 172.21.15.38:0/1600336752 shutdown_connections 2024-04-22T21:06:18.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.822+0000 7ff89eb88700 1 --2- 172.21.15.38:0/1600336752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8980fe810 0x7ff898106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:18.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.822+0000 7ff89eb88700 1 --2- 172.21.15.38:0/1600336752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8980fde70 0x7ff8980fe240 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:18.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.822+0000 7ff89eb88700 1 -- 172.21.15.38:0/1600336752 >> 172.21.15.38:0/1600336752 conn(0x7ff8980f7880 msgr2=0x7ff8980f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:18.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.822+0000 7ff89eb88700 1 -- 172.21.15.38:0/1600336752 shutdown_connections 2024-04-22T21:06:18.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.822+0000 7ff89eb88700 1 -- 172.21.15.38:0/1600336752 wait complete. 2024-04-22T21:06:18.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.823+0000 7ff89eb88700 1 Processor -- start 2024-04-22T21:06:18.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.823+0000 7ff89eb88700 1 -- start start 2024-04-22T21:06:18.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.823+0000 7ff89eb88700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8980fde70 0x7ff8980fbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:18.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.823+0000 7ff89eb88700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8980fe810 0x7ff8980fc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:18.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.823+0000 7ff89eb88700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8980fc770 con 0x7ff8980fe810 2024-04-22T21:06:18.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.823+0000 7ff89eb88700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8980fc8b0 con 0x7ff8980fde70 2024-04-22T21:06:18.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.823+0000 7ff897fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8980fe810 0x7ff8980fc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:18.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.823+0000 7ff897fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8980fe810 0x7ff8980fc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58612/0 (socket says 172.21.15.38:58612) 2024-04-22T21:06:18.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.823+0000 7ff897fff700 1 -- 172.21.15.38:0/1726264269 learned_addr learned my addr 172.21.15.38:0/1726264269 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.823+0000 7ff89c924700 1 --2- 172.21.15.38:0/1726264269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8980fde70 0x7ff8980fbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.823+0000 7ff897fff700 1 -- 172.21.15.38:0/1726264269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8980fde70 msgr2=0x7ff8980fbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.823+0000 7ff897fff700 1 --2- 172.21.15.38:0/1726264269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8980fde70 0x7ff8980fbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.823+0000 7ff897fff700 1 -- 172.21.15.38:0/1726264269 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff890009940 con 0x7ff8980fe810 2024-04-22T21:06:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.824+0000 7ff897fff700 1 --2- 172.21.15.38:0/1726264269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8980fe810 0x7ff8980fc1e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff88c002a80 tx=0x7ff88c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.824+0000 7ff8957fa700 1 -- 172.21.15.38:0/1726264269 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff88c007cc0 con 0x7ff8980fe810 2024-04-22T21:06:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.824+0000 7ff89eb88700 1 -- 172.21.15.38:0/1726264269 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff898119db0 con 0x7ff8980fe810 2024-04-22T21:06:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.824+0000 7ff8957fa700 1 -- 172.21.15.38:0/1726264269 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff88c00f040 con 0x7ff8980fe810 2024-04-22T21:06:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.824+0000 7ff8957fa700 1 -- 172.21.15.38:0/1726264269 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff88c00dac0 con 0x7ff8980fe810 2024-04-22T21:06:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.824+0000 7ff89eb88700 1 -- 172.21.15.38:0/1726264269 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff89811a1b0 con 0x7ff8980fe810 2024-04-22T21:06:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.825+0000 7ff89eb88700 1 -- 172.21.15.38:0/1726264269 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff898066720 con 0x7ff8980fe810 2024-04-22T21:06:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.825+0000 7ff8957fa700 1 -- 172.21.15.38:0/1726264269 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ff88c00dc20 con 0x7ff8980fe810 2024-04-22T21:06:18.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.826+0000 7ff8957fa700 1 --2- 172.21.15.38:0/1726264269 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff8800692a0 0x7ff88006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:18.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.826+0000 7ff8957fa700 1 -- 172.21.15.38:0/1726264269 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff88c086ac0 con 0x7ff8980fe810 2024-04-22T21:06:18.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.826+0000 7ff89c924700 1 --2- 172.21.15.38:0/1726264269 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff8800692a0 0x7ff88006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:18.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.826+0000 7ff89c924700 1 --2- 172.21.15.38:0/1726264269 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff8800692a0 0x7ff88006b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff89000b740 tx=0x7ff890004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:18.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:18.830+0000 7ff8957fa700 1 -- 172.21.15.38:0/1726264269 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff88c0530b0 con 0x7ff8980fe810 2024-04-22T21:06:19.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:19.025+0000 7ff89eb88700 1 -- 172.21.15.38:0/1726264269 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff89804bed0 con 0x7ff8980fe810 2024-04-22T21:06:19.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:19.026+0000 7ff8957fa700 1 -- 172.21.15.38:0/1726264269 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff88c0566d0 con 0x7ff8980fe810 2024-04-22T21:06:19.022 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T21:06:19.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:19.028+0000 7ff89eb88700 1 -- 172.21.15.38:0/1726264269 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff8800692a0 msgr2=0x7ff88006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:19.028+0000 7ff89eb88700 1 --2- 172.21.15.38:0/1726264269 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff8800692a0 0x7ff88006b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff89000b740 tx=0x7ff890004d10 comp rx=0 tx=0).stop 2024-04-22T21:06:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:19.028+0000 7ff89eb88700 1 -- 172.21.15.38:0/1726264269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8980fe810 msgr2=0x7ff8980fc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:19.028+0000 7ff89eb88700 1 --2- 172.21.15.38:0/1726264269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8980fe810 0x7ff8980fc1e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff88c002a80 tx=0x7ff88c0046b0 comp rx=0 tx=0).stop 2024-04-22T21:06:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:19.028+0000 7ff89eb88700 1 -- 172.21.15.38:0/1726264269 shutdown_connections 2024-04-22T21:06:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:19.028+0000 7ff89eb88700 1 --2- 172.21.15.38:0/1726264269 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff8800692a0 0x7ff88006b750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:19.028+0000 7ff89eb88700 1 --2- 172.21.15.38:0/1726264269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8980fde70 0x7ff8980fbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:19.028+0000 7ff89eb88700 1 --2- 172.21.15.38:0/1726264269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8980fe810 0x7ff8980fc1e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:19.028+0000 7ff89eb88700 1 -- 172.21.15.38:0/1726264269 >> 172.21.15.38:0/1726264269 conn(0x7ff8980f7880 msgr2=0x7ff8980f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:19.028+0000 7ff89eb88700 1 -- 172.21.15.38:0/1726264269 shutdown_connections 2024-04-22T21:06:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:19.028+0000 7ff89eb88700 1 -- 172.21.15.38:0/1726264269 wait complete. 2024-04-22T21:06:19.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:19 smithi098 ceph-mon[34053]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:19.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:19 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/1726264269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T21:06:19.577 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:19 smithi038 ceph-mon[26374]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:19.577 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:19 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1726264269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T21:06:19.602 INFO:tasks.cephadm:Setting up client nodes... 2024-04-22T21:06:19.602 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T21:06:19.865 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:21.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.327+0000 7fa21923d700 1 -- 172.21.15.38:0/2902409629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2140fc310 msgr2=0x7fa2140fc720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:21.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.327+0000 7fa21923d700 1 --2- 172.21.15.38:0/2902409629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2140fc310 0x7fa2140fc720 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa208004940 tx=0x7fa208009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:21.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.327+0000 7fa21923d700 1 -- 172.21.15.38:0/2902409629 shutdown_connections 2024-04-22T21:06:21.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.327+0000 7fa21923d700 1 --2- 172.21.15.38:0/2902409629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2140fcc60 0x7fa2140fd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:21.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.327+0000 7fa21923d700 1 --2- 172.21.15.38:0/2902409629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2140fc310 0x7fa2140fc720 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:21.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.327+0000 7fa21923d700 1 -- 172.21.15.38:0/2902409629 >> 172.21.15.38:0/2902409629 conn(0x7fa2140f6c00 msgr2=0x7fa2140f9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:21.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.328+0000 7fa21923d700 1 -- 172.21.15.38:0/2902409629 shutdown_connections 2024-04-22T21:06:21.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.328+0000 7fa21923d700 1 -- 172.21.15.38:0/2902409629 wait complete. 2024-04-22T21:06:21.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.328+0000 7fa21923d700 1 Processor -- start 2024-04-22T21:06:21.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.328+0000 7fa21923d700 1 -- start start 2024-04-22T21:06:21.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.329+0000 7fa21923d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2140fc310 0x7fa21410c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:21.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.329+0000 7fa21923d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2140fcc60 0x7fa21410cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:21.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.329+0000 7fa21923d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21410d3e0 con 0x7fa2140fc310 2024-04-22T21:06:21.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.329+0000 7fa21923d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21410d550 con 0x7fa2140fcc60 2024-04-22T21:06:21.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.329+0000 7fa212d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2140fc310 0x7fa21410c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:21.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.329+0000 7fa212d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2140fc310 0x7fa21410c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58626/0 (socket says 172.21.15.38:58626) 2024-04-22T21:06:21.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.329+0000 7fa212d9d700 1 -- 172.21.15.38:0/658114907 learned_addr learned my addr 172.21.15.38:0/658114907 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:21.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.329+0000 7fa21259c700 1 --2- 172.21.15.38:0/658114907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2140fcc60 0x7fa21410cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:21.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.329+0000 7fa212d9d700 1 -- 172.21.15.38:0/658114907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2140fcc60 msgr2=0x7fa21410cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:21.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.329+0000 7fa212d9d700 1 --2- 172.21.15.38:0/658114907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2140fcc60 0x7fa21410cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:21.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.329+0000 7fa212d9d700 1 -- 172.21.15.38:0/658114907 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa208005040 con 0x7fa2140fc310 2024-04-22T21:06:21.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.329+0000 7fa212d9d700 1 --2- 172.21.15.38:0/658114907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2140fc310 0x7fa21410c8a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa20800b3b0 tx=0x7fa208009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:21.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.329+0000 7fa1fb7fe700 1 -- 172.21.15.38:0/658114907 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa20800e040 con 0x7fa2140fc310 2024-04-22T21:06:21.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.329+0000 7fa1fb7fe700 1 -- 172.21.15.38:0/658114907 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa20800bbf0 con 0x7fa2140fc310 2024-04-22T21:06:21.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.329+0000 7fa21923d700 1 -- 172.21.15.38:0/658114907 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa214111f50 con 0x7fa2140fc310 2024-04-22T21:06:21.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.329+0000 7fa1fb7fe700 1 -- 172.21.15.38:0/658114907 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa20801c630 con 0x7fa2140fc310 2024-04-22T21:06:21.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.330+0000 7fa21923d700 1 -- 172.21.15.38:0/658114907 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa214112480 con 0x7fa2140fc310 2024-04-22T21:06:21.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.331+0000 7fa21923d700 1 -- 172.21.15.38:0/658114907 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa21404c710 con 0x7fa2140fc310 2024-04-22T21:06:21.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.331+0000 7fa1fb7fe700 1 -- 172.21.15.38:0/658114907 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fa20800bd60 con 0x7fa2140fc310 2024-04-22T21:06:21.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.331+0000 7fa1fb7fe700 1 --2- 172.21.15.38:0/658114907 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa1fc0692a0 0x7fa1fc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:21.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.331+0000 7fa1fb7fe700 1 -- 172.21.15.38:0/658114907 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa208087b00 con 0x7fa2140fc310 2024-04-22T21:06:21.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.331+0000 7fa21259c700 1 --2- 172.21.15.38:0/658114907 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa1fc0692a0 0x7fa1fc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:21.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.332+0000 7fa21259c700 1 --2- 172.21.15.38:0/658114907 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa1fc0692a0 0x7fa1fc06b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa204004f60 tx=0x7fa20400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:21.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.336+0000 7fa1fb7fe700 1 -- 172.21.15.38:0/658114907 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa208057c00 con 0x7fa2140fc310 2024-04-22T21:06:21.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:21 smithi038 ceph-mon[26374]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:21.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.612+0000 7fa21923d700 1 -- 172.21.15.38:0/658114907 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fa2141126a0 con 0x7fa2140fc310 2024-04-22T21:06:21.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.618+0000 7fa1fb7fe700 1 -- 172.21.15.38:0/658114907 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7fa208057790 con 0x7fa2140fc310 2024-04-22T21:06:21.614 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2024-04-22T21:06:21.614 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBN0SZmfWagJBAA+iOewreqBii72ccWVK0E2Q== 2024-04-22T21:06:21.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.620+0000 7fa21923d700 1 -- 172.21.15.38:0/658114907 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa1fc0692a0 msgr2=0x7fa1fc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:21.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.620+0000 7fa21923d700 1 --2- 172.21.15.38:0/658114907 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa1fc0692a0 0x7fa1fc06b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa204004f60 tx=0x7fa20400b040 comp rx=0 tx=0).stop 2024-04-22T21:06:21.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.620+0000 7fa21923d700 1 -- 172.21.15.38:0/658114907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2140fc310 msgr2=0x7fa21410c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:21.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.620+0000 7fa21923d700 1 --2- 172.21.15.38:0/658114907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2140fc310 0x7fa21410c8a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa20800b3b0 tx=0x7fa208009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:21.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.620+0000 7fa21923d700 1 -- 172.21.15.38:0/658114907 shutdown_connections 2024-04-22T21:06:21.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.620+0000 7fa21923d700 1 --2- 172.21.15.38:0/658114907 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa1fc0692a0 0x7fa1fc06b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:21.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.620+0000 7fa21923d700 1 --2- 172.21.15.38:0/658114907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2140fc310 0x7fa21410c8a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:21.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.620+0000 7fa21923d700 1 --2- 172.21.15.38:0/658114907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2140fcc60 0x7fa21410cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:21.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.620+0000 7fa21923d700 1 -- 172.21.15.38:0/658114907 >> 172.21.15.38:0/658114907 conn(0x7fa2140f6c00 msgr2=0x7fa2140f8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:21.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.620+0000 7fa21923d700 1 -- 172.21.15.38:0/658114907 shutdown_connections 2024-04-22T21:06:21.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:21.620+0000 7fa21923d700 1 -- 172.21.15.38:0/658114907 wait complete. 2024-04-22T21:06:21.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:21 smithi098 ceph-mon[34053]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:22.256 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T21:06:22.256 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-22T21:06:22.256 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T21:06:22.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T21:06:22.556 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi098/config 2024-04-22T21:06:22.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:22 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/658114907' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:06:22.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:22 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/658114907' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T21:06:22.679 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:22 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/658114907' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:06:22.679 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:22 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/658114907' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T21:06:23.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:23 smithi098 ceph-mon[34053]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:23.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:23 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:06:23.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:23 smithi038 ceph-mon[26374]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:23.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:23 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:06:24.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.227+0000 7f7f34feb700 1 -- 172.21.15.98:0/3740338627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f300fb480 msgr2=0x7f7f300fb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:24.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.227+0000 7f7f34feb700 1 --2- 172.21.15.98:0/3740338627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f300fb480 0x7f7f300fb890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7f20004a50 tx=0x7f7f2000af40 comp rx=0 tx=0).stop 2024-04-22T21:06:24.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.228+0000 7f7f34feb700 1 -- 172.21.15.98:0/3740338627 shutdown_connections 2024-04-22T21:06:24.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.228+0000 7f7f34feb700 1 --2- 172.21.15.98:0/3740338627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f300fc730 0x7f7f300fcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:24.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.228+0000 7f7f34feb700 1 --2- 172.21.15.98:0/3740338627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f300fb480 0x7f7f300fb890 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:24.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.228+0000 7f7f34feb700 1 -- 172.21.15.98:0/3740338627 >> 172.21.15.98:0/3740338627 conn(0x7f7f300f6be0 msgr2=0x7f7f300f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:24.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.228+0000 7f7f34feb700 1 -- 172.21.15.98:0/3740338627 shutdown_connections 2024-04-22T21:06:24.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.228+0000 7f7f34feb700 1 -- 172.21.15.98:0/3740338627 wait complete. 2024-04-22T21:06:24.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.228+0000 7f7f34feb700 1 Processor -- start 2024-04-22T21:06:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.228+0000 7f7f34feb700 1 -- start start 2024-04-22T21:06:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f34feb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f300fb480 0x7f7f3010b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f34feb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f300fc730 0x7f7f3010bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f34feb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3010c2f0 con 0x7f7f300fb480 2024-04-22T21:06:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f34feb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f30110c90 con 0x7f7f300fc730 2024-04-22T21:06:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f25d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f300fc730 0x7f7f3010bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f25d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f300fc730 0x7f7f3010bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36552/0 (socket says 172.21.15.98:36552) 2024-04-22T21:06:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f25d9b700 1 -- 172.21.15.98:0/2375218372 learned_addr learned my addr 172.21.15.98:0/2375218372 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-22T21:06:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f2e59c700 1 --2- 172.21.15.98:0/2375218372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f300fb480 0x7f7f3010b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f25d9b700 1 -- 172.21.15.98:0/2375218372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f300fb480 msgr2=0x7f7f3010b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f25d9b700 1 --2- 172.21.15.98:0/2375218372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f300fb480 0x7f7f3010b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:24.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f25d9b700 1 -- 172.21.15.98:0/2375218372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f20005040 con 0x7f7f300fc730 2024-04-22T21:06:24.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f25d9b700 1 --2- 172.21.15.98:0/2375218372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f300fc730 0x7f7f3010bc60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7f1800b790 tx=0x7f7f18009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:24.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f277fe700 1 -- 172.21.15.98:0/2375218372 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f1800f6f0 con 0x7f7f300fc730 2024-04-22T21:06:24.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f277fe700 1 -- 172.21.15.98:0/2375218372 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f1800fd20 con 0x7f7f300fc730 2024-04-22T21:06:24.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f277fe700 1 -- 172.21.15.98:0/2375218372 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f180103a0 con 0x7f7f300fc730 2024-04-22T21:06:24.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f34feb700 1 -- 172.21.15.98:0/2375218372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f30110ef0 con 0x7f7f300fc730 2024-04-22T21:06:24.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.229+0000 7f7f34feb700 1 -- 172.21.15.98:0/2375218372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f30111430 con 0x7f7f300fc730 2024-04-22T21:06:24.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.230+0000 7f7f34feb700 1 -- 172.21.15.98:0/2375218372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f3004c710 con 0x7f7f300fc730 2024-04-22T21:06:24.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.231+0000 7f7f277fe700 1 -- 172.21.15.98:0/2375218372 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f7f18011400 con 0x7f7f300fc730 2024-04-22T21:06:24.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.231+0000 7f7f277fe700 1 --2- 172.21.15.98:0/2375218372 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7f1c0692a0 0x7f7f1c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:24.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.231+0000 7f7f277fe700 1 -- 172.21.15.98:0/2375218372 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7f18086b00 con 0x7f7f300fc730 2024-04-22T21:06:24.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.232+0000 7f7f2e59c700 1 --2- 172.21.15.98:0/2375218372 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7f1c0692a0 0x7f7f1c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:24.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.232+0000 7f7f2e59c700 1 --2- 172.21.15.98:0/2375218372 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7f1c0692a0 0x7f7f1c06b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7f2000c3b0 tx=0x7f7f2000af40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:24.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.236+0000 7f7f277fe700 1 -- 172.21.15.98:0/2375218372 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f7f18056ca0 con 0x7f7f300fc730 2024-04-22T21:06:24.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.492+0000 7f7f34feb700 1 -- 172.21.15.98:0/2375218372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f7f30103760 con 0x7f7f300fc730 2024-04-22T21:06:24.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.498+0000 7f7f277fe700 1 -- 172.21.15.98:0/2375218372 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f7f1805ae60 con 0x7f7f300fc730 2024-04-22T21:06:24.494 INFO:teuthology.orchestra.run.smithi098.stdout:[client.1] 2024-04-22T21:06:24.494 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBQ0SZm5+57HRAALCPcKWoTLxpxtsLaCgKdNg== 2024-04-22T21:06:24.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.500+0000 7f7f34feb700 1 -- 172.21.15.98:0/2375218372 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7f1c0692a0 msgr2=0x7f7f1c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:24.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.500+0000 7f7f34feb700 1 --2- 172.21.15.98:0/2375218372 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7f1c0692a0 0x7f7f1c06b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7f2000c3b0 tx=0x7f7f2000af40 comp rx=0 tx=0).stop 2024-04-22T21:06:24.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.500+0000 7f7f34feb700 1 -- 172.21.15.98:0/2375218372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f300fc730 msgr2=0x7f7f3010bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:24.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.500+0000 7f7f34feb700 1 --2- 172.21.15.98:0/2375218372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f300fc730 0x7f7f3010bc60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7f1800b790 tx=0x7f7f18009250 comp rx=0 tx=0).stop 2024-04-22T21:06:24.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.500+0000 7f7f34feb700 1 -- 172.21.15.98:0/2375218372 shutdown_connections 2024-04-22T21:06:24.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.500+0000 7f7f34feb700 1 --2- 172.21.15.98:0/2375218372 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7f1c0692a0 0x7f7f1c06b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:24.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.500+0000 7f7f34feb700 1 --2- 172.21.15.98:0/2375218372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f300fb480 0x7f7f3010b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:24.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.500+0000 7f7f34feb700 1 --2- 172.21.15.98:0/2375218372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f300fc730 0x7f7f3010bc60 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:24.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.500+0000 7f7f34feb700 1 -- 172.21.15.98:0/2375218372 >> 172.21.15.98:0/2375218372 conn(0x7f7f300f6be0 msgr2=0x7f7f300f8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:24.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.500+0000 7f7f34feb700 1 -- 172.21.15.98:0/2375218372 shutdown_connections 2024-04-22T21:06:24.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:06:24.500+0000 7f7f34feb700 1 -- 172.21.15.98:0/2375218372 wait complete. 2024-04-22T21:06:25.114 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T21:06:25.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-22T21:06:25.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-22T21:06:25.149 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T21:06:25.149 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-22T21:06:25.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-22T21:06:25.425 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:25.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:25 smithi098 ceph-mon[34053]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:25.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:25 smithi098 ceph-mon[34053]: from='client.? 172.21.15.98:0/2375218372' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:06:25.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:25 smithi098 ceph-mon[34053]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:06:25.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:25 smithi098 ceph-mon[34053]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T21:06:25.547 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:25 smithi038 ceph-mon[26374]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:25.548 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:25 smithi038 ceph-mon[26374]: from='client.? 172.21.15.98:0/2375218372' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:06:25.548 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:25 smithi038 ceph-mon[26374]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:06:25.548 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:25 smithi038 ceph-mon[26374]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T21:06:26.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.972+0000 7f6a59f46700 1 -- 172.21.15.38:0/460796990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a540656b0 msgr2=0x7f6a54065b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.972+0000 7f6a59f46700 1 --2- 172.21.15.38:0/460796990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a540656b0 0x7f6a54065b20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6a48004940 tx=0x7f6a48009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.972+0000 7f6a59f46700 1 -- 172.21.15.38:0/460796990 shutdown_connections 2024-04-22T21:06:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.972+0000 7f6a59f46700 1 --2- 172.21.15.38:0/460796990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a540656b0 0x7f6a54065b20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.972+0000 7f6a59f46700 1 --2- 172.21.15.38:0/460796990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a54102e60 0x7f6a54103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.972+0000 7f6a59f46700 1 -- 172.21.15.38:0/460796990 >> 172.21.15.38:0/460796990 conn(0x7f6a54072bc0 msgr2=0x7f6a54072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.972+0000 7f6a59f46700 1 -- 172.21.15.38:0/460796990 shutdown_connections 2024-04-22T21:06:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.972+0000 7f6a59f46700 1 -- 172.21.15.38:0/460796990 wait complete. 2024-04-22T21:06:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.973+0000 7f6a59f46700 1 Processor -- start 2024-04-22T21:06:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.973+0000 7f6a59f46700 1 -- start start 2024-04-22T21:06:26.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.973+0000 7f6a59f46700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a540656b0 0x7f6a54113b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:26.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.973+0000 7f6a59f46700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a54102e60 0x7f6a54114070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:26.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.973+0000 7f6a59f46700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a541146c0 con 0x7f6a540656b0 2024-04-22T21:06:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.973+0000 7f6a59f46700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5410dab0 con 0x7f6a54102e60 2024-04-22T21:06:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.973+0000 7f6a537fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a540656b0 0x7f6a54113b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.973+0000 7f6a537fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a540656b0 0x7f6a54113b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59212/0 (socket says 172.21.15.38:59212) 2024-04-22T21:06:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.973+0000 7f6a537fe700 1 -- 172.21.15.38:0/1683182579 learned_addr learned my addr 172.21.15.38:0/1683182579 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.973+0000 7f6a537fe700 1 -- 172.21.15.38:0/1683182579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a54102e60 msgr2=0x7f6a54114070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.973+0000 7f6a537fe700 1 --2- 172.21.15.38:0/1683182579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a54102e60 0x7f6a54114070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.973+0000 7f6a537fe700 1 -- 172.21.15.38:0/1683182579 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a48005040 con 0x7f6a540656b0 2024-04-22T21:06:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.974+0000 7f6a537fe700 1 --2- 172.21.15.38:0/1683182579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a540656b0 0x7f6a54113b30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6a4400cb10 tx=0x7f6a44007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.974+0000 7f6a3bfff700 1 -- 172.21.15.38:0/1683182579 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a44004200 con 0x7f6a540656b0 2024-04-22T21:06:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.974+0000 7f6a3bfff700 1 -- 172.21.15.38:0/1683182579 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a4400acc0 con 0x7f6a540656b0 2024-04-22T21:06:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.974+0000 7f6a59f46700 1 -- 172.21.15.38:0/1683182579 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a5410dd90 con 0x7f6a540656b0 2024-04-22T21:06:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.974+0000 7f6a3bfff700 1 -- 172.21.15.38:0/1683182579 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a44003d90 con 0x7f6a540656b0 2024-04-22T21:06:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.974+0000 7f6a59f46700 1 -- 172.21.15.38:0/1683182579 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a5410e350 con 0x7f6a540656b0 2024-04-22T21:06:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.975+0000 7f6a59f46700 1 -- 172.21.15.38:0/1683182579 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a54100130 con 0x7f6a540656b0 2024-04-22T21:06:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.975+0000 7f6a3bfff700 1 -- 172.21.15.38:0/1683182579 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f6a44003430 con 0x7f6a540656b0 2024-04-22T21:06:26.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.976+0000 7f6a3bfff700 1 --2- 172.21.15.38:0/1683182579 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6a3c0692a0 0x7f6a3c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:26.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.976+0000 7f6a3bfff700 1 -- 172.21.15.38:0/1683182579 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6a44087460 con 0x7f6a540656b0 2024-04-22T21:06:26.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.976+0000 7f6a52ffd700 1 --2- 172.21.15.38:0/1683182579 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6a3c0692a0 0x7f6a3c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:26.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.976+0000 7f6a52ffd700 1 --2- 172.21.15.38:0/1683182579 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6a3c0692a0 0x7f6a3c06b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6a48004f90 tx=0x7f6a48009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:26.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:26.980+0000 7f6a3bfff700 1 -- 172.21.15.38:0/1683182579 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6a440574e0 con 0x7f6a540656b0 2024-04-22T21:06:27.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:27.233+0000 7f6a59f46700 1 -- 172.21.15.38:0/1683182579 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f6a5410f1b0 con 0x7f6a540656b0 2024-04-22T21:06:27.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:27.236+0000 7f6a3bfff700 1 -- 172.21.15.38:0/1683182579 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167167 (secure 0 0 0) 0x7f6a44057070 con 0x7f6a540656b0 2024-04-22T21:06:27.235 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:06:27.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:27.239+0000 7f6a59f46700 1 -- 172.21.15.38:0/1683182579 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6a3c0692a0 msgr2=0x7f6a3c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:27.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:27.239+0000 7f6a59f46700 1 --2- 172.21.15.38:0/1683182579 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6a3c0692a0 0x7f6a3c06b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6a48004f90 tx=0x7f6a48009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:27.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:27.239+0000 7f6a59f46700 1 -- 172.21.15.38:0/1683182579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a540656b0 msgr2=0x7f6a54113b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:27.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:27.239+0000 7f6a59f46700 1 --2- 172.21.15.38:0/1683182579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a540656b0 0x7f6a54113b30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6a4400cb10 tx=0x7f6a44007f90 comp rx=0 tx=0).stop 2024-04-22T21:06:27.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:27.239+0000 7f6a59f46700 1 -- 172.21.15.38:0/1683182579 shutdown_connections 2024-04-22T21:06:27.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:27.239+0000 7f6a59f46700 1 --2- 172.21.15.38:0/1683182579 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6a3c0692a0 0x7f6a3c06b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:27.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:27.239+0000 7f6a59f46700 1 --2- 172.21.15.38:0/1683182579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a540656b0 0x7f6a54113b30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:27.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:27.239+0000 7f6a59f46700 1 --2- 172.21.15.38:0/1683182579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a54102e60 0x7f6a54114070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:27.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:27.239+0000 7f6a59f46700 1 -- 172.21.15.38:0/1683182579 >> 172.21.15.38:0/1683182579 conn(0x7f6a54072bc0 msgr2=0x7f6a540faec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:27.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:27.239+0000 7f6a59f46700 1 -- 172.21.15.38:0/1683182579 shutdown_connections 2024-04-22T21:06:27.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:27.239+0000 7f6a59f46700 1 -- 172.21.15.38:0/1683182579 wait complete. 2024-04-22T21:06:27.560 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:27 smithi038 ceph-mon[26374]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:27.560 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:27 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1683182579' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T21:06:27.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:27 smithi098 ceph-mon[34053]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:27.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:27 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/1683182579' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T21:06:27.771 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":19,"flags":0,"active_gid":14212,"active_name":"smithi038.lxfgut","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":368620},{"type":"v1","addr":"172.21.15.38:6801","nonce":368620}]},"active_addr":"172.21.15.38:6801/368620","active_change":"2024-04-22T21:02:38.036876+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14232,"name":"smithi098.azqohf","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.38:8443/","prometheus":"http://172.21.15.38:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":1688398524}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":3319306200}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":2295462390}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":1349520153}]}]} 2024-04-22T21:06:27.775 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-22T21:06:27.775 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-22T21:06:27.775 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T21:06:28.041 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:29.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:29 smithi098 ceph-mon[34053]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:29.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.528+0000 7f78b5f6b700 1 -- 172.21.15.38:0/1520676502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f78b01026c0 msgr2=0x7f78b00fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:29.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.528+0000 7f78b5f6b700 1 --2- 172.21.15.38:0/1520676502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f78b01026c0 0x7f78b00fbd30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f78a4004940 tx=0x7f78a4009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:29.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.528+0000 7f78b5f6b700 1 -- 172.21.15.38:0/1520676502 shutdown_connections 2024-04-22T21:06:29.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.528+0000 7f78b5f6b700 1 --2- 172.21.15.38:0/1520676502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f78b01026c0 0x7f78b00fbd30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:29.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.528+0000 7f78b5f6b700 1 --2- 172.21.15.38:0/1520676502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b0101d20 0x7f78b01020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:29.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.528+0000 7f78b5f6b700 1 -- 172.21.15.38:0/1520676502 >> 172.21.15.38:0/1520676502 conn(0x7f78b00f7870 msgr2=0x7f78b00f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:29.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.528+0000 7f78b5f6b700 1 -- 172.21.15.38:0/1520676502 shutdown_connections 2024-04-22T21:06:29.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.528+0000 7f78b5f6b700 1 -- 172.21.15.38:0/1520676502 wait complete. 2024-04-22T21:06:29.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.529+0000 7f78b5f6b700 1 Processor -- start 2024-04-22T21:06:29.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.529+0000 7f78b5f6b700 1 -- start start 2024-04-22T21:06:29.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.529+0000 7f78b5f6b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f78b0101d20 0x7f78b0113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:29.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.529+0000 7f78b5f6b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b01026c0 0x7f78b0113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:29.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.529+0000 7f78b5f6b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b0114510 con 0x7f78b0101d20 2024-04-22T21:06:29.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.529+0000 7f78b5f6b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b010db10 con 0x7f78b01026c0 2024-04-22T21:06:29.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.529+0000 7f78af7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f78b0101d20 0x7f78b0113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:29.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.529+0000 7f78af7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f78b0101d20 0x7f78b0113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59230/0 (socket says 172.21.15.38:59230) 2024-04-22T21:06:29.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.529+0000 7f78af7fe700 1 -- 172.21.15.38:0/3729997518 learned_addr learned my addr 172.21.15.38:0/3729997518 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:29.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.530+0000 7f78aeffd700 1 --2- 172.21.15.38:0/3729997518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b01026c0 0x7f78b0113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:29.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.530+0000 7f78af7fe700 1 -- 172.21.15.38:0/3729997518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b01026c0 msgr2=0x7f78b0113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:29.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.530+0000 7f78af7fe700 1 --2- 172.21.15.38:0/3729997518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b01026c0 0x7f78b0113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:29.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.530+0000 7f78af7fe700 1 -- 172.21.15.38:0/3729997518 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a4005040 con 0x7f78b0101d20 2024-04-22T21:06:29.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.530+0000 7f78af7fe700 1 --2- 172.21.15.38:0/3729997518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f78b0101d20 0x7f78b0113980 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f78a000c830 tx=0x7f78a0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:29.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.530+0000 7f7897fff700 1 -- 172.21.15.38:0/3729997518 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78a000a760 con 0x7f78b0101d20 2024-04-22T21:06:29.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.530+0000 7f78b5f6b700 1 -- 172.21.15.38:0/3729997518 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b010ddf0 con 0x7f78b0101d20 2024-04-22T21:06:29.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.530+0000 7f7897fff700 1 -- 172.21.15.38:0/3729997518 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78a000ad90 con 0x7f78b0101d20 2024-04-22T21:06:29.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.530+0000 7f7897fff700 1 -- 172.21.15.38:0/3729997518 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78a00113a0 con 0x7f78b0101d20 2024-04-22T21:06:29.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.530+0000 7f78b5f6b700 1 -- 172.21.15.38:0/3729997518 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b010e3b0 con 0x7f78b0101d20 2024-04-22T21:06:29.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.531+0000 7f78b5f6b700 1 -- 172.21.15.38:0/3729997518 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b00fd470 con 0x7f78b0101d20 2024-04-22T21:06:29.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.531+0000 7f7897fff700 1 -- 172.21.15.38:0/3729997518 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f78a000a8c0 con 0x7f78b0101d20 2024-04-22T21:06:29.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.532+0000 7f7897fff700 1 --2- 172.21.15.38:0/3729997518 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f78980692a0 0x7f789806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:29.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.532+0000 7f7897fff700 1 -- 172.21.15.38:0/3729997518 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f78a0086ec0 con 0x7f78b0101d20 2024-04-22T21:06:29.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.532+0000 7f78aeffd700 1 --2- 172.21.15.38:0/3729997518 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f78980692a0 0x7f789806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:29.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.532+0000 7f78aeffd700 1 --2- 172.21.15.38:0/3729997518 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f78980692a0 0x7f789806b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f78b010ecd0 tx=0x7f78a4009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:29.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.536+0000 7f7897fff700 1 -- 172.21.15.38:0/3729997518 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f78a0056fc0 con 0x7f78b0101d20 2024-04-22T21:06:29.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:29 smithi038 ceph-mon[26374]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:29.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.730+0000 7f78b5f6b700 1 -- 172.21.15.38:0/3729997518 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78b0108e00 con 0x7f78b0101d20 2024-04-22T21:06:29.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.731+0000 7f7897fff700 1 -- 172.21.15.38:0/3729997518 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11022 (secure 0 0 0) 0x7f78a0056b50 con 0x7f78b0101d20 2024-04-22T21:06:29.727 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:06:29.727 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":41,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","created":"2024-04-22T21:00:25.280047+0000","modified":"2024-04-22T21:06:11.235267+0000","last_up_change":"2024-04-22T21:06:10.229585+0000","last_in_change":"2024-04-22T21:05:51.689258+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T21:04:56.557561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"aeb7d8ce-e081-4a6a-8c2e-560bcb819a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":454874338},{"type":"v1","addr":"172.21.15.38:6803","nonce":454874338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":454874338},{"type":"v1","addr":"172.21.15.38:6805","nonce":454874338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":454874338},{"type":"v1","addr":"172.21.15.38:6809","nonce":454874338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":454874338},{"type":"v1","addr":"172.21.15.38:6807","nonce":454874338}]},"public_addr":"172.21.15.38:6803/454874338","cluster_addr":"172.21.15.38:6805/454874338","heartbeat_back_addr":"172.21.15.38:6809/454874338","heartbeat_front_addr":"172.21.15.38:6807/454874338","state":["exists","up"]},{"osd":1,"uuid":"7c90956f-6cb0-4a71-86c0-9c38e9710808","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1336148940},{"type":"v1","addr":"172.21.15.38:6811","nonce":1336148940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1336148940},{"type":"v1","addr":"172.21.15.38:6813","nonce":1336148940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1336148940},{"type":"v1","addr":"172.21.15.38:6817","nonce":1336148940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1336148940},{"type":"v1","addr":"172.21.15.38:6815","nonce":1336148940}]},"public_addr":"172.21.15.38:6811/1336148940","cluster_addr":"172.21.15.38:6813/1336148940","heartbeat_back_addr":"172.21.15.38:6817/1336148940","heartbeat_front_addr":"172.21.15.38:6815/1336148940","state":["exists","up"]},{"osd":2,"uuid":"379616b9-bdc7-4fd6-a1c5-b937e2ef3202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":47489714},{"type":"v1","addr":"172.21.15.38:6819","nonce":47489714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":47489714},{"type":"v1","addr":"172.21.15.38:6821","nonce":47489714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":47489714},{"type":"v1","addr":"172.21.15.38:6825","nonce":47489714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":47489714},{"type":"v1","addr":"172.21.15.38:6823","nonce":47489714}]},"public_addr":"172.21.15.38:6819/47489714","cluster_addr":"172.21.15.38:6821/47489714","heartbeat_back_addr":"172.21.15.38:6825/47489714","heartbeat_front_addr":"172.21.15.38:6823/47489714","state":["exists","up"]},{"osd":3,"uuid":"becb1602-d471-4c38-8cd1-781828c4b2f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2538219727},{"type":"v1","addr":"172.21.15.98:6801","nonce":2538219727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2538219727},{"type":"v1","addr":"172.21.15.98:6803","nonce":2538219727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2538219727},{"type":"v1","addr":"172.21.15.98:6807","nonce":2538219727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2538219727},{"type":"v1","addr":"172.21.15.98:6805","nonce":2538219727}]},"public_addr":"172.21.15.98:6801/2538219727","cluster_addr":"172.21.15.98:6803/2538219727","heartbeat_back_addr":"172.21.15.98:6807/2538219727","heartbeat_front_addr":"172.21.15.98:6805/2538219727","state":["exists","up"]},{"osd":4,"uuid":"c886eb15-63f1-4bb9-a229-0278c8372f5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3945015359},{"type":"v1","addr":"172.21.15.98:6809","nonce":3945015359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3945015359},{"type":"v1","addr":"172.21.15.98:6811","nonce":3945015359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3945015359},{"type":"v1","addr":"172.21.15.98:6815","nonce":3945015359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3945015359},{"type":"v1","addr":"172.21.15.98:6813","nonce":3945015359}]},"public_addr":"172.21.15.98:6809/3945015359","cluster_addr":"172.21.15.98:6811/3945015359","heartbeat_back_addr":"172.21.15.98:6815/3945015359","heartbeat_front_addr":"172.21.15.98:6813/3945015359","state":["exists","up"]},{"osd":5,"uuid":"58e5fcf4-d0cc-4498-ae06-d51d8f2d0933","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2442541230},{"type":"v1","addr":"172.21.15.98:6817","nonce":2442541230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2442541230},{"type":"v1","addr":"172.21.15.98:6819","nonce":2442541230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2442541230},{"type":"v1","addr":"172.21.15.98:6823","nonce":2442541230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2442541230},{"type":"v1","addr":"172.21.15.98:6821","nonce":2442541230}]},"public_addr":"172.21.15.98:6817/2442541230","cluster_addr":"172.21.15.98:6819/2442541230","heartbeat_back_addr":"172.21.15.98:6823/2442541230","heartbeat_front_addr":"172.21.15.98:6821/2442541230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:03:53.821993+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:04:24.335225+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:04:52.351848+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:05:17.370839+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:05:43.310131+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:06:09.074139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/3596300465":"2024-04-23T21:02:38.036748+0000","172.21.15.38:6801/989134174":"2024-04-23T21:02:38.036748+0000","172.21.15.38:6800/989134174":"2024-04-23T21:02:38.036748+0000","172.21.15.38:0/420277838":"2024-04-23T21:02:38.036748+0000","172.21.15.38:0/3486303720":"2024-04-23T21:01:31.045112+0000","172.21.15.38:6800/2551561659":"2024-04-23T21:01:31.045112+0000","172.21.15.38:0/2835639508":"2024-04-23T21:01:31.045112+0000","172.21.15.38:6801/2787760150":"2024-04-23T21:00:59.025532+0000","172.21.15.38:0/3765130717":"2024-04-23T21:00:59.025532+0000","172.21.15.38:6801/2551561659":"2024-04-23T21:01:31.045112+0000","172.21.15.38:6800/2787760150":"2024-04-23T21:00:59.025532+0000","172.21.15.38:0/2942485563":"2024-04-23T21:00:59.025532+0000","172.21.15.38:0/4050800451":"2024-04-23T21:02:38.036748+0000","172.21.15.38:0/2754521346":"2024-04-23T21:01:31.045112+0000","172.21.15.38:0/3342218679":"2024-04-23T21:00:59.025532+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T21:06:29.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.733+0000 7f78b5f6b700 1 -- 172.21.15.38:0/3729997518 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f78980692a0 msgr2=0x7f789806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:29.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.733+0000 7f78b5f6b700 1 --2- 172.21.15.38:0/3729997518 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f78980692a0 0x7f789806b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f78b010ecd0 tx=0x7f78a4009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:29.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.733+0000 7f78b5f6b700 1 -- 172.21.15.38:0/3729997518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f78b0101d20 msgr2=0x7f78b0113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:29.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.733+0000 7f78b5f6b700 1 --2- 172.21.15.38:0/3729997518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f78b0101d20 0x7f78b0113980 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f78a000c830 tx=0x7f78a0004470 comp rx=0 tx=0).stop 2024-04-22T21:06:29.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.733+0000 7f78b5f6b700 1 -- 172.21.15.38:0/3729997518 shutdown_connections 2024-04-22T21:06:29.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.733+0000 7f78b5f6b700 1 --2- 172.21.15.38:0/3729997518 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f78980692a0 0x7f789806b750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:29.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.733+0000 7f78b5f6b700 1 --2- 172.21.15.38:0/3729997518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f78b0101d20 0x7f78b0113980 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:29.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.733+0000 7f78b5f6b700 1 --2- 172.21.15.38:0/3729997518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b01026c0 0x7f78b0113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:29.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.733+0000 7f78b5f6b700 1 -- 172.21.15.38:0/3729997518 >> 172.21.15.38:0/3729997518 conn(0x7f78b00f7870 msgr2=0x7f78b00f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:29.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.733+0000 7f78b5f6b700 1 -- 172.21.15.38:0/3729997518 shutdown_connections 2024-04-22T21:06:29.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:29.733+0000 7f78b5f6b700 1 -- 172.21.15.38:0/3729997518 wait complete. 2024-04-22T21:06:30.317 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-22T21:06:30.318 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T21:06:30.586 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:30.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:30 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3729997518' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:06:30.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:30 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/3729997518' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:06:31.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:31 smithi038 ceph-mon[26374]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:31.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:31 smithi098 ceph-mon[34053]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:32.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.033+0000 7fb5a6b45700 1 -- 172.21.15.38:0/2744491398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5a01026b0 msgr2=0x7fb5a00fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:32.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.033+0000 7fb5a6b45700 1 --2- 172.21.15.38:0/2744491398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5a01026b0 0x7fb5a00fbd20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb58c004940 tx=0x7fb58c009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:32.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.033+0000 7fb5a6b45700 1 -- 172.21.15.38:0/2744491398 shutdown_connections 2024-04-22T21:06:32.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.033+0000 7fb5a6b45700 1 --2- 172.21.15.38:0/2744491398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5a01026b0 0x7fb5a00fbd20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:32.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.033+0000 7fb5a6b45700 1 --2- 172.21.15.38:0/2744491398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5a0101d10 0x7fb5a01020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:32.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.033+0000 7fb5a6b45700 1 -- 172.21.15.38:0/2744491398 >> 172.21.15.38:0/2744491398 conn(0x7fb5a0072bc0 msgr2=0x7fb5a0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:32.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.033+0000 7fb5a6b45700 1 -- 172.21.15.38:0/2744491398 shutdown_connections 2024-04-22T21:06:32.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.033+0000 7fb5a6b45700 1 -- 172.21.15.38:0/2744491398 wait complete. 2024-04-22T21:06:32.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.034+0000 7fb5a6b45700 1 Processor -- start 2024-04-22T21:06:32.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.034+0000 7fb5a6b45700 1 -- start start 2024-04-22T21:06:32.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.034+0000 7fb5a6b45700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5a0101d10 0x7fb5a00fd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:32.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.034+0000 7fb5a6b45700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5a01026b0 0x7fb5a00fded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:32.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.034+0000 7fb5a6b45700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a00fe460 con 0x7fb5a0101d10 2024-04-22T21:06:32.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.034+0000 7fb5a6b45700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a00fe5a0 con 0x7fb5a01026b0 2024-04-22T21:06:32.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.034+0000 7fb5a48e1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5a0101d10 0x7fb5a00fd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:32.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.034+0000 7fb5a48e1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5a0101d10 0x7fb5a00fd990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59246/0 (socket says 172.21.15.38:59246) 2024-04-22T21:06:32.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.034+0000 7fb5a48e1700 1 -- 172.21.15.38:0/1676049158 learned_addr learned my addr 172.21.15.38:0/1676049158 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:32.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.034+0000 7fb59ffff700 1 --2- 172.21.15.38:0/1676049158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5a01026b0 0x7fb5a00fded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:32.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.035+0000 7fb5a48e1700 1 -- 172.21.15.38:0/1676049158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5a01026b0 msgr2=0x7fb5a00fded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:32.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.035+0000 7fb5a48e1700 1 --2- 172.21.15.38:0/1676049158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5a01026b0 0x7fb5a00fded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:32.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.035+0000 7fb5a48e1700 1 -- 172.21.15.38:0/1676049158 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58c005040 con 0x7fb5a0101d10 2024-04-22T21:06:32.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.035+0000 7fb5a48e1700 1 --2- 172.21.15.38:0/1676049158 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5a0101d10 0x7fb5a00fd990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb594002a50 tx=0x7fb5940046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:32.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.035+0000 7fb59d7fa700 1 -- 172.21.15.38:0/1676049158 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb594007cc0 con 0x7fb5a0101d10 2024-04-22T21:06:32.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.035+0000 7fb5a6b45700 1 -- 172.21.15.38:0/1676049158 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a0119e80 con 0x7fb5a0101d10 2024-04-22T21:06:32.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.035+0000 7fb59d7fa700 1 -- 172.21.15.38:0/1676049158 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb59400f040 con 0x7fb5a0101d10 2024-04-22T21:06:32.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.035+0000 7fb59d7fa700 1 -- 172.21.15.38:0/1676049158 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb59400da30 con 0x7fb5a0101d10 2024-04-22T21:06:32.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.035+0000 7fb5a6b45700 1 -- 172.21.15.38:0/1676049158 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a011a2a0 con 0x7fb5a0101d10 2024-04-22T21:06:32.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.036+0000 7fb5a6b45700 1 -- 172.21.15.38:0/1676049158 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a00fd460 con 0x7fb5a0101d10 2024-04-22T21:06:32.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.036+0000 7fb59d7fa700 1 -- 172.21.15.38:0/1676049158 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fb59400db90 con 0x7fb5a0101d10 2024-04-22T21:06:32.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.037+0000 7fb59d7fa700 1 --2- 172.21.15.38:0/1676049158 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fb5900692a0 0x7fb59006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:32.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.037+0000 7fb59d7fa700 1 -- 172.21.15.38:0/1676049158 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb594086b80 con 0x7fb5a0101d10 2024-04-22T21:06:32.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.037+0000 7fb59ffff700 1 --2- 172.21.15.38:0/1676049158 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fb5900692a0 0x7fb59006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:32.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.037+0000 7fb59ffff700 1 --2- 172.21.15.38:0/1676049158 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fb5900692a0 0x7fb59006b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb58c004890 tx=0x7fb58c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:32.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.041+0000 7fb59d7fa700 1 -- 172.21.15.38:0/1676049158 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb594053170 con 0x7fb5a0101d10 2024-04-22T21:06:32.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.237+0000 7fb5a6b45700 1 -- 172.21.15.38:0/1676049158 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5a0108e50 con 0x7fb5a0101d10 2024-04-22T21:06:32.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.238+0000 7fb59d7fa700 1 -- 172.21.15.38:0/1676049158 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11022 (secure 0 0 0) 0x7fb594056790 con 0x7fb5a0101d10 2024-04-22T21:06:32.234 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:06:32.235 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":41,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","created":"2024-04-22T21:00:25.280047+0000","modified":"2024-04-22T21:06:11.235267+0000","last_up_change":"2024-04-22T21:06:10.229585+0000","last_in_change":"2024-04-22T21:05:51.689258+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T21:04:56.557561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"aeb7d8ce-e081-4a6a-8c2e-560bcb819a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":454874338},{"type":"v1","addr":"172.21.15.38:6803","nonce":454874338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":454874338},{"type":"v1","addr":"172.21.15.38:6805","nonce":454874338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":454874338},{"type":"v1","addr":"172.21.15.38:6809","nonce":454874338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":454874338},{"type":"v1","addr":"172.21.15.38:6807","nonce":454874338}]},"public_addr":"172.21.15.38:6803/454874338","cluster_addr":"172.21.15.38:6805/454874338","heartbeat_back_addr":"172.21.15.38:6809/454874338","heartbeat_front_addr":"172.21.15.38:6807/454874338","state":["exists","up"]},{"osd":1,"uuid":"7c90956f-6cb0-4a71-86c0-9c38e9710808","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1336148940},{"type":"v1","addr":"172.21.15.38:6811","nonce":1336148940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1336148940},{"type":"v1","addr":"172.21.15.38:6813","nonce":1336148940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1336148940},{"type":"v1","addr":"172.21.15.38:6817","nonce":1336148940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1336148940},{"type":"v1","addr":"172.21.15.38:6815","nonce":1336148940}]},"public_addr":"172.21.15.38:6811/1336148940","cluster_addr":"172.21.15.38:6813/1336148940","heartbeat_back_addr":"172.21.15.38:6817/1336148940","heartbeat_front_addr":"172.21.15.38:6815/1336148940","state":["exists","up"]},{"osd":2,"uuid":"379616b9-bdc7-4fd6-a1c5-b937e2ef3202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":47489714},{"type":"v1","addr":"172.21.15.38:6819","nonce":47489714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":47489714},{"type":"v1","addr":"172.21.15.38:6821","nonce":47489714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":47489714},{"type":"v1","addr":"172.21.15.38:6825","nonce":47489714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":47489714},{"type":"v1","addr":"172.21.15.38:6823","nonce":47489714}]},"public_addr":"172.21.15.38:6819/47489714","cluster_addr":"172.21.15.38:6821/47489714","heartbeat_back_addr":"172.21.15.38:6825/47489714","heartbeat_front_addr":"172.21.15.38:6823/47489714","state":["exists","up"]},{"osd":3,"uuid":"becb1602-d471-4c38-8cd1-781828c4b2f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2538219727},{"type":"v1","addr":"172.21.15.98:6801","nonce":2538219727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2538219727},{"type":"v1","addr":"172.21.15.98:6803","nonce":2538219727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2538219727},{"type":"v1","addr":"172.21.15.98:6807","nonce":2538219727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2538219727},{"type":"v1","addr":"172.21.15.98:6805","nonce":2538219727}]},"public_addr":"172.21.15.98:6801/2538219727","cluster_addr":"172.21.15.98:6803/2538219727","heartbeat_back_addr":"172.21.15.98:6807/2538219727","heartbeat_front_addr":"172.21.15.98:6805/2538219727","state":["exists","up"]},{"osd":4,"uuid":"c886eb15-63f1-4bb9-a229-0278c8372f5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3945015359},{"type":"v1","addr":"172.21.15.98:6809","nonce":3945015359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3945015359},{"type":"v1","addr":"172.21.15.98:6811","nonce":3945015359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3945015359},{"type":"v1","addr":"172.21.15.98:6815","nonce":3945015359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3945015359},{"type":"v1","addr":"172.21.15.98:6813","nonce":3945015359}]},"public_addr":"172.21.15.98:6809/3945015359","cluster_addr":"172.21.15.98:6811/3945015359","heartbeat_back_addr":"172.21.15.98:6815/3945015359","heartbeat_front_addr":"172.21.15.98:6813/3945015359","state":["exists","up"]},{"osd":5,"uuid":"58e5fcf4-d0cc-4498-ae06-d51d8f2d0933","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2442541230},{"type":"v1","addr":"172.21.15.98:6817","nonce":2442541230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2442541230},{"type":"v1","addr":"172.21.15.98:6819","nonce":2442541230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2442541230},{"type":"v1","addr":"172.21.15.98:6823","nonce":2442541230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2442541230},{"type":"v1","addr":"172.21.15.98:6821","nonce":2442541230}]},"public_addr":"172.21.15.98:6817/2442541230","cluster_addr":"172.21.15.98:6819/2442541230","heartbeat_back_addr":"172.21.15.98:6823/2442541230","heartbeat_front_addr":"172.21.15.98:6821/2442541230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:03:53.821993+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:04:24.335225+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:04:52.351848+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:05:17.370839+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:05:43.310131+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:06:09.074139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/3596300465":"2024-04-23T21:02:38.036748+0000","172.21.15.38:6801/989134174":"2024-04-23T21:02:38.036748+0000","172.21.15.38:6800/989134174":"2024-04-23T21:02:38.036748+0000","172.21.15.38:0/420277838":"2024-04-23T21:02:38.036748+0000","172.21.15.38:0/3486303720":"2024-04-23T21:01:31.045112+0000","172.21.15.38:6800/2551561659":"2024-04-23T21:01:31.045112+0000","172.21.15.38:0/2835639508":"2024-04-23T21:01:31.045112+0000","172.21.15.38:6801/2787760150":"2024-04-23T21:00:59.025532+0000","172.21.15.38:0/3765130717":"2024-04-23T21:00:59.025532+0000","172.21.15.38:6801/2551561659":"2024-04-23T21:01:31.045112+0000","172.21.15.38:6800/2787760150":"2024-04-23T21:00:59.025532+0000","172.21.15.38:0/2942485563":"2024-04-23T21:00:59.025532+0000","172.21.15.38:0/4050800451":"2024-04-23T21:02:38.036748+0000","172.21.15.38:0/2754521346":"2024-04-23T21:01:31.045112+0000","172.21.15.38:0/3342218679":"2024-04-23T21:00:59.025532+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T21:06:32.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.240+0000 7fb5a6b45700 1 -- 172.21.15.38:0/1676049158 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fb5900692a0 msgr2=0x7fb59006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:32.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.240+0000 7fb5a6b45700 1 --2- 172.21.15.38:0/1676049158 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fb5900692a0 0x7fb59006b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb58c004890 tx=0x7fb58c009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:32.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.240+0000 7fb5a6b45700 1 -- 172.21.15.38:0/1676049158 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5a0101d10 msgr2=0x7fb5a00fd990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:32.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.240+0000 7fb5a6b45700 1 --2- 172.21.15.38:0/1676049158 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5a0101d10 0x7fb5a00fd990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb594002a50 tx=0x7fb5940046b0 comp rx=0 tx=0).stop 2024-04-22T21:06:32.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.240+0000 7fb5a6b45700 1 -- 172.21.15.38:0/1676049158 shutdown_connections 2024-04-22T21:06:32.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.240+0000 7fb5a6b45700 1 --2- 172.21.15.38:0/1676049158 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fb5900692a0 0x7fb59006b750 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:32.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.240+0000 7fb5a6b45700 1 --2- 172.21.15.38:0/1676049158 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5a0101d10 0x7fb5a00fd990 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:32.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.240+0000 7fb5a6b45700 1 --2- 172.21.15.38:0/1676049158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5a01026b0 0x7fb5a00fded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:32.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.240+0000 7fb5a6b45700 1 -- 172.21.15.38:0/1676049158 >> 172.21.15.38:0/1676049158 conn(0x7fb5a0072bc0 msgr2=0x7fb5a00fa840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:32.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.241+0000 7fb5a6b45700 1 -- 172.21.15.38:0/1676049158 shutdown_connections 2024-04-22T21:06:32.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:32.241+0000 7fb5a6b45700 1 -- 172.21.15.38:0/1676049158 wait complete. 2024-04-22T21:06:32.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:32 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1676049158' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:06:32.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:32 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/1676049158' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:06:32.935 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-22T21:06:32.936 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-22T21:06:32.937 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-22T21:06:32.937 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-22T21:06:32.938 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-22T21:06:32.938 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-22T21:06:33.253 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:33.254 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:33.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:33 smithi038 ceph-mon[26374]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:33.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:33 smithi098 ceph-mon[34053]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:34.306 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:34.309 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:34.514 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:34.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.706+0000 7f8c0f14d700 1 -- 172.21.15.38:0/1317984244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08102e60 msgr2=0x7f8c08103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:34.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.706+0000 7f8c0f14d700 1 --2- 172.21.15.38:0/1317984244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08102e60 0x7f8c08103230 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8009ce0 tx=0x7f8bf8004d10 comp rx=0 tx=0).stop 2024-04-22T21:06:34.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.706+0000 7f8c0f14d700 1 -- 172.21.15.38:0/1317984244 shutdown_connections 2024-04-22T21:06:34.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.706+0000 7f8c0f14d700 1 --2- 172.21.15.38:0/1317984244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c080656b0 0x7f8c08065b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:34.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.706+0000 7f8c0f14d700 1 --2- 172.21.15.38:0/1317984244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08102e60 0x7f8c08103230 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:34.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.706+0000 7f8c0f14d700 1 -- 172.21.15.38:0/1317984244 >> 172.21.15.38:0/1317984244 conn(0x7f8c08072bc0 msgr2=0x7f8c08072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:34.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.707+0000 7f8c0f14d700 1 -- 172.21.15.38:0/1317984244 shutdown_connections 2024-04-22T21:06:34.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.707+0000 7f8c0f14d700 1 -- 172.21.15.38:0/1317984244 wait complete. 2024-04-22T21:06:34.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.707+0000 7f8c0f14d700 1 Processor -- start 2024-04-22T21:06:34.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.707+0000 7f8c0f14d700 1 -- start start 2024-04-22T21:06:34.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.707+0000 7f8c0f14d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c080656b0 0x7f8c080fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:34.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.707+0000 7f8c0f14d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08102e60 0x7f8c080fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:34.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.707+0000 7f8c0f14d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c080fc8b0 con 0x7f8c08102e60 2024-04-22T21:06:34.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.707+0000 7f8c0f14d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c080fc9f0 con 0x7f8c080656b0 2024-04-22T21:06:34.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.708+0000 7f8c07fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08102e60 0x7f8c080fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:34.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.708+0000 7f8c07fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08102e60 0x7f8c080fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36766/0 (socket says 172.21.15.38:36766) 2024-04-22T21:06:34.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.708+0000 7f8c07fff700 1 -- 172.21.15.38:0/2198891711 learned_addr learned my addr 172.21.15.38:0/2198891711 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:34.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.708+0000 7f8c0cee9700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c080656b0 0x7f8c080fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:34.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.708+0000 7f8c07fff700 1 -- 172.21.15.38:0/2198891711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c080656b0 msgr2=0x7f8c080fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:34.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.708+0000 7f8c07fff700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c080656b0 0x7f8c080fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:34.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.708+0000 7f8c07fff700 1 -- 172.21.15.38:0/2198891711 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf8009940 con 0x7f8c08102e60 2024-04-22T21:06:34.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.708+0000 7f8c0cee9700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c080656b0 0x7f8c080fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T21:06:34.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.708+0000 7f8c07fff700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08102e60 0x7f8c080fc290 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000b790 tx=0x7f8bf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:34.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.708+0000 7f8c057fa700 1 -- 172.21.15.38:0/2198891711 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bf000f770 con 0x7f8c08102e60 2024-04-22T21:06:34.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.708+0000 7f8c057fa700 1 -- 172.21.15.38:0/2198891711 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8bf000fda0 con 0x7f8c08102e60 2024-04-22T21:06:34.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.708+0000 7f8c0f14d700 1 -- 172.21.15.38:0/2198891711 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0811a000 con 0x7f8c08102e60 2024-04-22T21:06:34.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.708+0000 7f8c057fa700 1 -- 172.21.15.38:0/2198891711 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bf000d310 con 0x7f8c08102e60 2024-04-22T21:06:34.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.708+0000 7f8c0f14d700 1 -- 172.21.15.38:0/2198891711 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0811a510 con 0x7f8c08102e60 2024-04-22T21:06:34.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.709+0000 7f8c0f14d700 1 -- 172.21.15.38:0/2198891711 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8c080faaf0 con 0x7f8c08102e60 2024-04-22T21:06:34.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.710+0000 7f8c057fa700 1 -- 172.21.15.38:0/2198891711 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f8bf0010420 con 0x7f8c08102e60 2024-04-22T21:06:34.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.710+0000 7f8c057fa700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8bf40692a0 0x7f8bf406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:34.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.710+0000 7f8c057fa700 1 -- 172.21.15.38:0/2198891711 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8bf0087850 con 0x7f8c08102e60 2024-04-22T21:06:34.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.710+0000 7f8c0cee9700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8bf40692a0 0x7f8bf406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:34.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.711+0000 7f8c057fa700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.98:6808/3945015359,v1:172.21.15.98:6809/3945015359] conn(0x7f8bf406ef70 0x7f8bf4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:34.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.711+0000 7f8c057fa700 1 -- 172.21.15.38:0/2198891711 --> [v2:172.21.15.98:6808/3945015359,v1:172.21.15.98:6809/3945015359] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8bf4071a30 con 0x7f8bf406ef70 2024-04-22T21:06:34.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.711+0000 7f8c057fa700 1 -- 172.21.15.38:0/2198891711 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8bf0087c50 con 0x7f8c08102e60 2024-04-22T21:06:34.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.711+0000 7f8c0cee9700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8bf40692a0 0x7f8bf406b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8bf80048c0 tx=0x7f8bf8004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:34.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.711+0000 7f8c0d6ea700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.98:6808/3945015359,v1:172.21.15.98:6809/3945015359] conn(0x7f8bf406ef70 0x7f8bf4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:34.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.711+0000 7f8c0d6ea700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.98:6808/3945015359,v1:172.21.15.98:6809/3945015359] conn(0x7f8bf406ef70 0x7f8bf4071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:34.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.712+0000 7f8c057fa700 1 -- 172.21.15.38:0/2198891711 <== osd.4 v2:172.21.15.98:6808/3945015359 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f8bf4071a30 con 0x7f8bf406ef70 2024-04-22T21:06:34.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.727+0000 7f8c0f14d700 1 -- 172.21.15.38:0/2198891711 --> [v2:172.21.15.98:6808/3945015359,v1:172.21.15.98:6809/3945015359] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8c080fcb80 con 0x7f8bf406ef70 2024-04-22T21:06:34.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.727+0000 7f8c057fa700 1 -- 172.21.15.38:0/2198891711 <== osd.4 v2:172.21.15.98:6808/3945015359 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8c00009980 con 0x7f8bf406ef70 2024-04-22T21:06:34.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.728+0000 7f8c0f14d700 1 -- 172.21.15.38:0/2198891711 >> [v2:172.21.15.98:6808/3945015359,v1:172.21.15.98:6809/3945015359] conn(0x7f8bf406ef70 msgr2=0x7f8bf4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:34.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.728+0000 7f8c0f14d700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.98:6808/3945015359,v1:172.21.15.98:6809/3945015359] conn(0x7f8bf406ef70 0x7f8bf4071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:34.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.728+0000 7f8c0f14d700 1 -- 172.21.15.38:0/2198891711 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8bf40692a0 msgr2=0x7f8bf406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:34.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.728+0000 7f8c0f14d700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8bf40692a0 0x7f8bf406b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8bf80048c0 tx=0x7f8bf8004d10 comp rx=0 tx=0).stop 2024-04-22T21:06:34.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.728+0000 7f8c0f14d700 1 -- 172.21.15.38:0/2198891711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08102e60 msgr2=0x7f8c080fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:34.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.728+0000 7f8c0f14d700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08102e60 0x7f8c080fc290 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000b790 tx=0x7f8bf0009250 comp rx=0 tx=0).stop 2024-04-22T21:06:34.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.728+0000 7f8c0f14d700 1 -- 172.21.15.38:0/2198891711 shutdown_connections 2024-04-22T21:06:34.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.728+0000 7f8c0f14d700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.98:6808/3945015359,v1:172.21.15.98:6809/3945015359] conn(0x7f8bf406ef70 0x7f8bf4071380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:34.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.728+0000 7f8c0f14d700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8bf40692a0 0x7f8bf406b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:34.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.728+0000 7f8c0f14d700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c080656b0 0x7f8c080fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:34.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.728+0000 7f8c0f14d700 1 --2- 172.21.15.38:0/2198891711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08102e60 0x7f8c080fc290 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:34.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.728+0000 7f8c0f14d700 1 -- 172.21.15.38:0/2198891711 >> 172.21.15.38:0/2198891711 conn(0x7f8c08072bc0 msgr2=0x7f8c080fa0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:34.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.728+0000 7f8c0f14d700 1 -- 172.21.15.38:0/2198891711 shutdown_connections 2024-04-22T21:06:34.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:34.728+0000 7f8c0f14d700 1 -- 172.21.15.38:0/2198891711 wait complete. 2024-04-22T21:06:35.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:35 smithi038 ceph-mon[26374]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:35.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:35 smithi098 ceph-mon[34053]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:36.094 INFO:teuthology.orchestra.run.smithi038.stdout:150323855372 2024-04-22T21:06:36.094 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-22T21:06:37.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:37 smithi038 ceph-mon[26374]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:37.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:37 smithi098 ceph-mon[34053]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:38.429 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:06:38.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:38 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:06:38.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.846+0000 7f8db86f6700 1 -- 172.21.15.38:0/1609119986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8db01037c0 msgr2=0x7f8db00fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:38.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.846+0000 7f8db86f6700 1 --2- 172.21.15.38:0/1609119986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8db01037c0 0x7f8db00fce30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8da0004940 tx=0x7f8da0009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:38.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.846+0000 7f8db86f6700 1 -- 172.21.15.38:0/1609119986 shutdown_connections 2024-04-22T21:06:38.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.846+0000 7f8db86f6700 1 --2- 172.21.15.38:0/1609119986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8db01037c0 0x7f8db00fce30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:38.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.846+0000 7f8db86f6700 1 --2- 172.21.15.38:0/1609119986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8db0102e20 0x7f8db01031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:38.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.846+0000 7f8db86f6700 1 -- 172.21.15.38:0/1609119986 >> 172.21.15.38:0/1609119986 conn(0x7f8db00f89b0 msgr2=0x7f8db00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:38.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.846+0000 7f8db86f6700 1 -- 172.21.15.38:0/1609119986 shutdown_connections 2024-04-22T21:06:38.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.846+0000 7f8db86f6700 1 -- 172.21.15.38:0/1609119986 wait complete. 2024-04-22T21:06:38.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.846+0000 7f8db86f6700 1 Processor -- start 2024-04-22T21:06:38.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.847+0000 7f8db86f6700 1 -- start start 2024-04-22T21:06:38.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.847+0000 7f8db86f6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8db0102e20 0x7f8db01139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:38.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.847+0000 7f8db86f6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8db01037c0 0x7f8db0113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:38.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.847+0000 7f8db86f6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db0114530 con 0x7f8db01037c0 2024-04-22T21:06:38.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.847+0000 7f8db86f6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db010db50 con 0x7f8db0102e20 2024-04-22T21:06:38.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.847+0000 7f8db5c91700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8db01037c0 0x7f8db0113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:38.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.847+0000 7f8db5c91700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8db01037c0 0x7f8db0113ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36782/0 (socket says 172.21.15.38:36782) 2024-04-22T21:06:38.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.847+0000 7f8db5c91700 1 -- 172.21.15.38:0/4162608054 learned_addr learned my addr 172.21.15.38:0/4162608054 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:38.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.847+0000 7f8db6492700 1 --2- 172.21.15.38:0/4162608054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8db0102e20 0x7f8db01139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:38.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.847+0000 7f8db5c91700 1 -- 172.21.15.38:0/4162608054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8db0102e20 msgr2=0x7f8db01139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:38.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.847+0000 7f8db5c91700 1 --2- 172.21.15.38:0/4162608054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8db0102e20 0x7f8db01139a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:38.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.847+0000 7f8db5c91700 1 -- 172.21.15.38:0/4162608054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da0005040 con 0x7f8db01037c0 2024-04-22T21:06:38.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.847+0000 7f8db5c91700 1 --2- 172.21.15.38:0/4162608054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8db01037c0 0x7f8db0113ee0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009100 tx=0x7f8da0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:38.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.847+0000 7f8da6ffd700 1 -- 172.21.15.38:0/4162608054 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8da000e040 con 0x7f8db01037c0 2024-04-22T21:06:38.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.848+0000 7f8db86f6700 1 -- 172.21.15.38:0/4162608054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db010ddd0 con 0x7f8db01037c0 2024-04-22T21:06:38.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.848+0000 7f8db86f6700 1 -- 172.21.15.38:0/4162608054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db010e390 con 0x7f8db01037c0 2024-04-22T21:06:38.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.848+0000 7f8da6ffd700 1 -- 172.21.15.38:0/4162608054 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8da0013420 con 0x7f8db01037c0 2024-04-22T21:06:38.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.848+0000 7f8da6ffd700 1 -- 172.21.15.38:0/4162608054 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8da001c610 con 0x7f8db01037c0 2024-04-22T21:06:38.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.848+0000 7f8db86f6700 1 -- 172.21.15.38:0/4162608054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8db010e980 con 0x7f8db01037c0 2024-04-22T21:06:38.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.849+0000 7f8da6ffd700 1 -- 172.21.15.38:0/4162608054 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f8da0013590 con 0x7f8db01037c0 2024-04-22T21:06:38.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.849+0000 7f8da6ffd700 1 --2- 172.21.15.38:0/4162608054 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8d9c0692a0 0x7f8d9c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:38.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.850+0000 7f8da6ffd700 1 -- 172.21.15.38:0/4162608054 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8da0088a70 con 0x7f8db01037c0 2024-04-22T21:06:38.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.850+0000 7f8db6492700 1 --2- 172.21.15.38:0/4162608054 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8d9c0692a0 0x7f8d9c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:38.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.850+0000 7f8da6ffd700 1 --2- 172.21.15.38:0/4162608054 >> [v2:172.21.15.98:6800/2538219727,v1:172.21.15.98:6801/2538219727] conn(0x7f8d9c06ef70 0x7f8d9c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:38.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.850+0000 7f8da6ffd700 1 -- 172.21.15.38:0/4162608054 --> [v2:172.21.15.98:6800/2538219727,v1:172.21.15.98:6801/2538219727] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d9c071a30 con 0x7f8d9c06ef70 2024-04-22T21:06:38.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.850+0000 7f8da6ffd700 1 -- 172.21.15.38:0/4162608054 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8da0088e70 con 0x7f8db01037c0 2024-04-22T21:06:38.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.850+0000 7f8db6492700 1 --2- 172.21.15.38:0/4162608054 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8d9c0692a0 0x7f8d9c06b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8dac004550 tx=0x7f8dac009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:38.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.850+0000 7f8db6c93700 1 --2- 172.21.15.38:0/4162608054 >> [v2:172.21.15.98:6800/2538219727,v1:172.21.15.98:6801/2538219727] conn(0x7f8d9c06ef70 0x7f8d9c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:38.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.851+0000 7f8db6c93700 1 --2- 172.21.15.38:0/4162608054 >> [v2:172.21.15.98:6800/2538219727,v1:172.21.15.98:6801/2538219727] conn(0x7f8d9c06ef70 0x7f8d9c071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:38.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.851+0000 7f8da6ffd700 1 -- 172.21.15.38:0/4162608054 <== osd.3 v2:172.21.15.98:6800/2538219727 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f8d9c071a30 con 0x7f8d9c06ef70 2024-04-22T21:06:38.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.866+0000 7f8db86f6700 1 -- 172.21.15.38:0/4162608054 --> [v2:172.21.15.98:6800/2538219727,v1:172.21.15.98:6801/2538219727] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8db005e640 con 0x7f8d9c06ef70 2024-04-22T21:06:38.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.867+0000 7f8da6ffd700 1 -- 172.21.15.38:0/4162608054 <== osd.3 v2:172.21.15.98:6800/2538219727 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8db005e640 con 0x7f8d9c06ef70 2024-04-22T21:06:38.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.867+0000 7f8db86f6700 1 -- 172.21.15.38:0/4162608054 >> [v2:172.21.15.98:6800/2538219727,v1:172.21.15.98:6801/2538219727] conn(0x7f8d9c06ef70 msgr2=0x7f8d9c071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:38.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.867+0000 7f8db86f6700 1 --2- 172.21.15.38:0/4162608054 >> [v2:172.21.15.98:6800/2538219727,v1:172.21.15.98:6801/2538219727] conn(0x7f8d9c06ef70 0x7f8d9c071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:38.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.867+0000 7f8db86f6700 1 -- 172.21.15.38:0/4162608054 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8d9c0692a0 msgr2=0x7f8d9c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:38.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.867+0000 7f8db86f6700 1 --2- 172.21.15.38:0/4162608054 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8d9c0692a0 0x7f8d9c06b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8dac004550 tx=0x7f8dac009040 comp rx=0 tx=0).stop 2024-04-22T21:06:38.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.867+0000 7f8db86f6700 1 -- 172.21.15.38:0/4162608054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8db01037c0 msgr2=0x7f8db0113ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:38.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.867+0000 7f8db86f6700 1 --2- 172.21.15.38:0/4162608054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8db01037c0 0x7f8db0113ee0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009100 tx=0x7f8da0009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:38.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.867+0000 7f8db86f6700 1 -- 172.21.15.38:0/4162608054 shutdown_connections 2024-04-22T21:06:38.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.867+0000 7f8db86f6700 1 --2- 172.21.15.38:0/4162608054 >> [v2:172.21.15.98:6800/2538219727,v1:172.21.15.98:6801/2538219727] conn(0x7f8d9c06ef70 0x7f8d9c071380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:38.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.867+0000 7f8db86f6700 1 --2- 172.21.15.38:0/4162608054 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8d9c0692a0 0x7f8d9c06b750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:38.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.867+0000 7f8db86f6700 1 --2- 172.21.15.38:0/4162608054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8db0102e20 0x7f8db01139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:38.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.867+0000 7f8db86f6700 1 --2- 172.21.15.38:0/4162608054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8db01037c0 0x7f8db0113ee0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:38.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.867+0000 7f8db86f6700 1 -- 172.21.15.38:0/4162608054 >> 172.21.15.38:0/4162608054 conn(0x7f8db00f89b0 msgr2=0x7f8db01021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:38.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.867+0000 7f8db86f6700 1 -- 172.21.15.38:0/4162608054 shutdown_connections 2024-04-22T21:06:38.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:38.867+0000 7f8db86f6700 1 -- 172.21.15.38:0/4162608054 wait complete. 2024-04-22T21:06:39.493 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:39 smithi038 ceph-mon[26374]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:39.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:39 smithi098 ceph-mon[34053]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:40.885 INFO:teuthology.orchestra.run.smithi038.stdout:115964117010 2024-04-22T21:06:40.885 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-22T21:06:41.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.047+0000 7f048edc5700 1 -- 172.21.15.38:0/2361620223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04881037c0 msgr2=0x7f04880fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:41.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.047+0000 7f048edc5700 1 --2- 172.21.15.38:0/2361620223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04881037c0 0x7f04880fce30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0474004940 tx=0x7f0474009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:41.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.047+0000 7f048edc5700 1 -- 172.21.15.38:0/2361620223 shutdown_connections 2024-04-22T21:06:41.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.047+0000 7f048edc5700 1 --2- 172.21.15.38:0/2361620223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04881037c0 0x7f04880fce30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.047+0000 7f048edc5700 1 --2- 172.21.15.38:0/2361620223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0488102e20 0x7f04881031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.047+0000 7f048edc5700 1 -- 172.21.15.38:0/2361620223 >> 172.21.15.38:0/2361620223 conn(0x7f04880f8970 msgr2=0x7f04880fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:41.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.047+0000 7f048edc5700 1 -- 172.21.15.38:0/2361620223 shutdown_connections 2024-04-22T21:06:41.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.047+0000 7f048edc5700 1 -- 172.21.15.38:0/2361620223 wait complete. 2024-04-22T21:06:41.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.048+0000 7f048edc5700 1 Processor -- start 2024-04-22T21:06:41.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.048+0000 7f048edc5700 1 -- start start 2024-04-22T21:06:41.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.049+0000 7f048edc5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0488102e20 0x7f0488113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:41.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.049+0000 7f048edc5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04881037c0 0x7f0488114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:41.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.049+0000 7f048edc5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0488114670 con 0x7f04881037c0 2024-04-22T21:06:41.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.049+0000 7f048edc5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048810dab0 con 0x7f0488102e20 2024-04-22T21:06:41.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.049+0000 7f0487fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04881037c0 0x7f0488114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:41.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.049+0000 7f0487fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04881037c0 0x7f0488114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36824/0 (socket says 172.21.15.38:36824) 2024-04-22T21:06:41.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.049+0000 7f0487fff700 1 -- 172.21.15.38:0/4179363960 learned_addr learned my addr 172.21.15.38:0/4179363960 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:41.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.049+0000 7f048cb61700 1 --2- 172.21.15.38:0/4179363960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0488102e20 0x7f0488113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:41.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.050+0000 7f048cb61700 1 -- 172.21.15.38:0/4179363960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04881037c0 msgr2=0x7f0488114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:41.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.050+0000 7f048cb61700 1 --2- 172.21.15.38:0/4179363960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04881037c0 0x7f0488114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.050+0000 7f048cb61700 1 -- 172.21.15.38:0/4179363960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0474005040 con 0x7f0488102e20 2024-04-22T21:06:41.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.050+0000 7f048cb61700 1 --2- 172.21.15.38:0/4179363960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0488102e20 0x7f0488113ae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f047c009fd0 tx=0x7f047c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:41.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.050+0000 7f04857fa700 1 -- 172.21.15.38:0/4179363960 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f047c004e90 con 0x7f0488102e20 2024-04-22T21:06:41.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.051+0000 7f04857fa700 1 -- 172.21.15.38:0/4179363960 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f047c00f040 con 0x7f0488102e20 2024-04-22T21:06:41.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.051+0000 7f048edc5700 1 -- 172.21.15.38:0/4179363960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048810dd30 con 0x7f0488102e20 2024-04-22T21:06:41.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.051+0000 7f048edc5700 1 -- 172.21.15.38:0/4179363960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048810e270 con 0x7f0488102e20 2024-04-22T21:06:41.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.051+0000 7f04857fa700 1 -- 172.21.15.38:0/4179363960 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f047c00ca50 con 0x7f0488102e20 2024-04-22T21:06:41.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.052+0000 7f048edc5700 1 -- 172.21.15.38:0/4179363960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f048810e7a0 con 0x7f0488102e20 2024-04-22T21:06:41.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.052+0000 7f04857fa700 1 -- 172.21.15.38:0/4179363960 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f047c00cbb0 con 0x7f0488102e20 2024-04-22T21:06:41.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.053+0000 7f04857fa700 1 --2- 172.21.15.38:0/4179363960 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f04780692a0 0x7f047806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:41.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.053+0000 7f04857fa700 1 -- 172.21.15.38:0/4179363960 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f047c086ab0 con 0x7f0488102e20 2024-04-22T21:06:41.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.053+0000 7f0487fff700 1 --2- 172.21.15.38:0/4179363960 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f04780692a0 0x7f047806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:41.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.054+0000 7f04857fa700 1 --2- 172.21.15.38:0/4179363960 >> [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338] conn(0x7f047806ee00 0x7f0478071210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:41.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.054+0000 7f04857fa700 1 -- 172.21.15.38:0/4179363960 --> [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04780718c0 con 0x7f047806ee00 2024-04-22T21:06:41.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.054+0000 7f04857fa700 1 -- 172.21.15.38:0/4179363960 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f047c086df0 con 0x7f0488102e20 2024-04-22T21:06:41.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.054+0000 7f048d362700 1 --2- 172.21.15.38:0/4179363960 >> [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338] conn(0x7f047806ee00 0x7f0478071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:41.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.055+0000 7f048d362700 1 --2- 172.21.15.38:0/4179363960 >> [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338] conn(0x7f047806ee00 0x7f0478071210 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:41.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.056+0000 7f0487fff700 1 --2- 172.21.15.38:0/4179363960 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f04780692a0 0x7f047806b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0474007f50 tx=0x7f0474009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:41.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.056+0000 7f04857fa700 1 -- 172.21.15.38:0/4179363960 <== osd.0 v2:172.21.15.38:6802/454874338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f04780718c0 con 0x7f047806ee00 2024-04-22T21:06:41.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.078+0000 7f048edc5700 1 -- 172.21.15.38:0/4179363960 --> [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f048805e640 con 0x7f047806ee00 2024-04-22T21:06:41.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.078+0000 7f04857fa700 1 -- 172.21.15.38:0/4179363960 <== osd.0 v2:172.21.15.38:6802/454874338 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f048805e640 con 0x7f047806ee00 2024-04-22T21:06:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.079+0000 7f048edc5700 1 -- 172.21.15.38:0/4179363960 >> [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338] conn(0x7f047806ee00 msgr2=0x7f0478071210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.079+0000 7f048edc5700 1 --2- 172.21.15.38:0/4179363960 >> [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338] conn(0x7f047806ee00 0x7f0478071210 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.079+0000 7f048edc5700 1 -- 172.21.15.38:0/4179363960 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f04780692a0 msgr2=0x7f047806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.079+0000 7f048edc5700 1 --2- 172.21.15.38:0/4179363960 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f04780692a0 0x7f047806b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0474007f50 tx=0x7f0474009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.079+0000 7f048edc5700 1 -- 172.21.15.38:0/4179363960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0488102e20 msgr2=0x7f0488113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.079+0000 7f048edc5700 1 --2- 172.21.15.38:0/4179363960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0488102e20 0x7f0488113ae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f047c009fd0 tx=0x7f047c00b040 comp rx=0 tx=0).stop 2024-04-22T21:06:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.079+0000 7f048edc5700 1 -- 172.21.15.38:0/4179363960 shutdown_connections 2024-04-22T21:06:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.079+0000 7f048edc5700 1 --2- 172.21.15.38:0/4179363960 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f04780692a0 0x7f047806b750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.079+0000 7f048edc5700 1 --2- 172.21.15.38:0/4179363960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0488102e20 0x7f0488113ae0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.079+0000 7f048edc5700 1 --2- 172.21.15.38:0/4179363960 >> [v2:172.21.15.38:6802/454874338,v1:172.21.15.38:6803/454874338] conn(0x7f047806ee00 0x7f0478071210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.079+0000 7f048edc5700 1 --2- 172.21.15.38:0/4179363960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04881037c0 0x7f0488114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.079+0000 7f048edc5700 1 -- 172.21.15.38:0/4179363960 >> 172.21.15.38:0/4179363960 conn(0x7f04880f8970 msgr2=0x7f0488102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.079+0000 7f048edc5700 1 -- 172.21.15.38:0/4179363960 shutdown_connections 2024-04-22T21:06:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.079+0000 7f048edc5700 1 -- 172.21.15.38:0/4179363960 wait complete. 2024-04-22T21:06:41.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.910+0000 7fa4e0c6c700 1 -- 172.21.15.38:0/3397591514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4dc0fbbd0 msgr2=0x7fa4dc0fbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:41.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.910+0000 7fa4e0c6c700 1 --2- 172.21.15.38:0/3397591514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4dc0fbbd0 0x7fa4dc0fbfa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc004940 tx=0x7fa4cc009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:41.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.910+0000 7fa4e0c6c700 1 -- 172.21.15.38:0/3397591514 shutdown_connections 2024-04-22T21:06:41.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.910+0000 7fa4e0c6c700 1 --2- 172.21.15.38:0/3397591514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4dc0fc4e0 0x7fa4dc109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.910+0000 7fa4e0c6c700 1 --2- 172.21.15.38:0/3397591514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4dc0fbbd0 0x7fa4dc0fbfa0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.910+0000 7fa4e0c6c700 1 -- 172.21.15.38:0/3397591514 >> 172.21.15.38:0/3397591514 conn(0x7fa4dc0f7880 msgr2=0x7fa4dc0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:41.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.911+0000 7fa4e0c6c700 1 -- 172.21.15.38:0/3397591514 shutdown_connections 2024-04-22T21:06:41.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.911+0000 7fa4e0c6c700 1 -- 172.21.15.38:0/3397591514 wait complete. 2024-04-22T21:06:41.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.911+0000 7fa4e0c6c700 1 Processor -- start 2024-04-22T21:06:41.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.911+0000 7fa4e0c6c700 1 -- start start 2024-04-22T21:06:41.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.911+0000 7fa4e0c6c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4dc0fbbd0 0x7fa4dc115c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:41.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.911+0000 7fa4e0c6c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4dc0fc4e0 0x7fa4dc116170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.911+0000 7fa4e0c6c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc1167c0 con 0x7fa4dc0fbbd0 2024-04-22T21:06:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.911+0000 7fa4e0c6c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc10fbd0 con 0x7fa4dc0fc4e0 2024-04-22T21:06:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.911+0000 7fa4da59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4dc0fbbd0 0x7fa4dc115c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.911+0000 7fa4da59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4dc0fbbd0 0x7fa4dc115c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36800/0 (socket says 172.21.15.38:36800) 2024-04-22T21:06:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.912+0000 7fa4da59c700 1 -- 172.21.15.38:0/620624740 learned_addr learned my addr 172.21.15.38:0/620624740 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.912+0000 7fa4d9d9b700 1 --2- 172.21.15.38:0/620624740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4dc0fc4e0 0x7fa4dc116170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.912+0000 7fa4da59c700 1 -- 172.21.15.38:0/620624740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4dc0fc4e0 msgr2=0x7fa4dc116170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.912+0000 7fa4da59c700 1 --2- 172.21.15.38:0/620624740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4dc0fc4e0 0x7fa4dc116170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.912+0000 7fa4da59c700 1 -- 172.21.15.38:0/620624740 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc005040 con 0x7fa4dc0fbbd0 2024-04-22T21:06:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.912+0000 7fa4da59c700 1 --2- 172.21.15.38:0/620624740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4dc0fbbd0 0x7fa4dc115c30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009380 tx=0x7fa4cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.912+0000 7fa4caffd700 1 -- 172.21.15.38:0/620624740 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4cc00e040 con 0x7fa4dc0fbbd0 2024-04-22T21:06:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.912+0000 7fa4e0c6c700 1 -- 172.21.15.38:0/620624740 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4dc10feb0 con 0x7fa4dc0fbbd0 2024-04-22T21:06:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.912+0000 7fa4caffd700 1 -- 172.21.15.38:0/620624740 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4cc009af0 con 0x7fa4dc0fbbd0 2024-04-22T21:06:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.912+0000 7fa4caffd700 1 -- 172.21.15.38:0/620624740 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4cc01c3f0 con 0x7fa4dc0fbbd0 2024-04-22T21:06:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.912+0000 7fa4e0c6c700 1 -- 172.21.15.38:0/620624740 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4dc110470 con 0x7fa4dc0fbbd0 2024-04-22T21:06:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.913+0000 7fa4e0c6c700 1 -- 172.21.15.38:0/620624740 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa4dc110990 con 0x7fa4dc0fbbd0 2024-04-22T21:06:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.914+0000 7fa4caffd700 1 -- 172.21.15.38:0/620624740 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fa4cc013420 con 0x7fa4dc0fbbd0 2024-04-22T21:06:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.914+0000 7fa4caffd700 1 --2- 172.21.15.38:0/620624740 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa4c40692a0 0x7fa4c406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.914+0000 7fa4caffd700 1 -- 172.21.15.38:0/620624740 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa4cc0881e0 con 0x7fa4dc0fbbd0 2024-04-22T21:06:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.914+0000 7fa4d9d9b700 1 --2- 172.21.15.38:0/620624740 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa4c40692a0 0x7fa4c406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.914+0000 7fa4caffd700 1 --2- 172.21.15.38:0/620624740 >> [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714] conn(0x7fa4c406ef70 0x7fa4c4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.914+0000 7fa4caffd700 1 -- 172.21.15.38:0/620624740 --> [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4c4071a30 con 0x7fa4c406ef70 2024-04-22T21:06:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.914+0000 7fa4caffd700 1 -- 172.21.15.38:0/620624740 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4cc08a830 con 0x7fa4dc0fbbd0 2024-04-22T21:06:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.915+0000 7fa4dad9d700 1 --2- 172.21.15.38:0/620624740 >> [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714] conn(0x7fa4c406ef70 0x7fa4c4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.915+0000 7fa4d9d9b700 1 --2- 172.21.15.38:0/620624740 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa4c40692a0 0x7fa4c406b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc111510 tx=0x7fa4d000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.915+0000 7fa4dad9d700 1 --2- 172.21.15.38:0/620624740 >> [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714] conn(0x7fa4c406ef70 0x7fa4c4071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.916+0000 7fa4caffd700 1 -- 172.21.15.38:0/620624740 <== osd.2 v2:172.21.15.38:6818/47489714 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa4c4071a30 con 0x7fa4c406ef70 2024-04-22T21:06:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.933+0000 7fa4e0c6c700 1 -- 172.21.15.38:0/620624740 --> [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa4dc066720 con 0x7fa4c406ef70 2024-04-22T21:06:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.933+0000 7fa4caffd700 1 -- 172.21.15.38:0/620624740 <== osd.2 v2:172.21.15.38:6818/47489714 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa4dc066720 con 0x7fa4c406ef70 2024-04-22T21:06:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.934+0000 7fa4e0c6c700 1 -- 172.21.15.38:0/620624740 >> [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714] conn(0x7fa4c406ef70 msgr2=0x7fa4c4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.934+0000 7fa4e0c6c700 1 --2- 172.21.15.38:0/620624740 >> [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714] conn(0x7fa4c406ef70 0x7fa4c4071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.934+0000 7fa4e0c6c700 1 -- 172.21.15.38:0/620624740 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa4c40692a0 msgr2=0x7fa4c406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.934+0000 7fa4e0c6c700 1 --2- 172.21.15.38:0/620624740 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa4c40692a0 0x7fa4c406b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc111510 tx=0x7fa4d000b040 comp rx=0 tx=0).stop 2024-04-22T21:06:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.934+0000 7fa4e0c6c700 1 -- 172.21.15.38:0/620624740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4dc0fbbd0 msgr2=0x7fa4dc115c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.934+0000 7fa4e0c6c700 1 --2- 172.21.15.38:0/620624740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4dc0fbbd0 0x7fa4dc115c30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009380 tx=0x7fa4cc009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.934+0000 7fa4e0c6c700 1 -- 172.21.15.38:0/620624740 shutdown_connections 2024-04-22T21:06:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.934+0000 7fa4e0c6c700 1 --2- 172.21.15.38:0/620624740 >> [v2:172.21.15.38:6818/47489714,v1:172.21.15.38:6819/47489714] conn(0x7fa4c406ef70 0x7fa4c4071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.934+0000 7fa4e0c6c700 1 --2- 172.21.15.38:0/620624740 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa4c40692a0 0x7fa4c406b750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.934+0000 7fa4e0c6c700 1 --2- 172.21.15.38:0/620624740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4dc0fbbd0 0x7fa4dc115c30 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.934+0000 7fa4e0c6c700 1 --2- 172.21.15.38:0/620624740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4dc0fc4e0 0x7fa4dc116170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.934+0000 7fa4e0c6c700 1 -- 172.21.15.38:0/620624740 >> 172.21.15.38:0/620624740 conn(0x7fa4dc0f7880 msgr2=0x7fa4dc0f7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.934+0000 7fa4e0c6c700 1 -- 172.21.15.38:0/620624740 shutdown_connections 2024-04-22T21:06:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:40.934+0000 7fa4e0c6c700 1 -- 172.21.15.38:0/620624740 wait complete. 2024-04-22T21:06:41.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.574+0000 7f6f05fc1700 1 -- 172.21.15.38:0/1136033801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f00102e20 msgr2=0x7f6f001031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:41.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.574+0000 7f6f05fc1700 1 --2- 172.21.15.38:0/1136033801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f00102e20 0x7f6f001031f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0004940 tx=0x7f6ef0009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:41.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.574+0000 7f6f05fc1700 1 -- 172.21.15.38:0/1136033801 shutdown_connections 2024-04-22T21:06:41.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.574+0000 7f6f05fc1700 1 --2- 172.21.15.38:0/1136033801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f001037c0 0x7f6f000fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.574+0000 7f6f05fc1700 1 --2- 172.21.15.38:0/1136033801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f00102e20 0x7f6f001031f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.574+0000 7f6f05fc1700 1 -- 172.21.15.38:0/1136033801 >> 172.21.15.38:0/1136033801 conn(0x7f6f000f89b0 msgr2=0x7f6f000fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:41.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.575+0000 7f6f05fc1700 1 -- 172.21.15.38:0/1136033801 shutdown_connections 2024-04-22T21:06:41.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.575+0000 7f6f05fc1700 1 -- 172.21.15.38:0/1136033801 wait complete. 2024-04-22T21:06:41.572 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:41 smithi038 ceph-mon[26374]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:41.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.575+0000 7f6f05fc1700 1 Processor -- start 2024-04-22T21:06:41.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.575+0000 7f6f05fc1700 1 -- start start 2024-04-22T21:06:41.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.576+0000 7f6f05fc1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f00102e20 0x7f6f00115b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:41.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.576+0000 7f6f05fc1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f001037c0 0x7f6f001160c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:41.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.576+0000 7f6f05fc1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f001167a0 con 0x7f6f001037c0 2024-04-22T21:06:41.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.576+0000 7f6f05fc1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0010fcf0 con 0x7f6f00102e20 2024-04-22T21:06:41.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.576+0000 7f6efeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f001037c0 0x7f6f001160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:41.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.576+0000 7f6efeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f001037c0 0x7f6f001160c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36838/0 (socket says 172.21.15.38:36838) 2024-04-22T21:06:41.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.576+0000 7f6efeffd700 1 -- 172.21.15.38:0/1541555294 learned_addr learned my addr 172.21.15.38:0/1541555294 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:41.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.576+0000 7f6eff7fe700 1 --2- 172.21.15.38:0/1541555294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f00102e20 0x7f6f00115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:41.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.576+0000 7f6efeffd700 1 -- 172.21.15.38:0/1541555294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f00102e20 msgr2=0x7f6f00115b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:41.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.576+0000 7f6efeffd700 1 --2- 172.21.15.38:0/1541555294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f00102e20 0x7f6f00115b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.576+0000 7f6efeffd700 1 -- 172.21.15.38:0/1541555294 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef0005040 con 0x7f6f001037c0 2024-04-22T21:06:41.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.576+0000 7f6efeffd700 1 --2- 172.21.15.38:0/1541555294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f001037c0 0x7f6f001160c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400b790 tx=0x7f6ef4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:41.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.577+0000 7f6ee7fff700 1 -- 172.21.15.38:0/1541555294 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ef400f6f0 con 0x7f6f001037c0 2024-04-22T21:06:41.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.577+0000 7f6ee7fff700 1 -- 172.21.15.38:0/1541555294 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ef400fd20 con 0x7f6f001037c0 2024-04-22T21:06:41.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.577+0000 7f6ee7fff700 1 -- 172.21.15.38:0/1541555294 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ef40103a0 con 0x7f6f001037c0 2024-04-22T21:06:41.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.577+0000 7f6f05fc1700 1 -- 172.21.15.38:0/1541555294 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0010ff70 con 0x7f6f001037c0 2024-04-22T21:06:41.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.577+0000 7f6f05fc1700 1 -- 172.21.15.38:0/1541555294 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f00110530 con 0x7f6f001037c0 2024-04-22T21:06:41.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.578+0000 7f6f05fc1700 1 -- 172.21.15.38:0/1541555294 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6f00110ac0 con 0x7f6f001037c0 2024-04-22T21:06:41.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.578+0000 7f6ee7fff700 1 -- 172.21.15.38:0/1541555294 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f6ef4011400 con 0x7f6f001037c0 2024-04-22T21:06:41.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.579+0000 7f6ee7fff700 1 --2- 172.21.15.38:0/1541555294 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6ee80692a0 0x7f6ee806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:41.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.579+0000 7f6ee7fff700 1 -- 172.21.15.38:0/1541555294 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6ef4087510 con 0x7f6f001037c0 2024-04-22T21:06:41.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.579+0000 7f6eff7fe700 1 --2- 172.21.15.38:0/1541555294 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6ee80692a0 0x7f6ee806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:41.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.579+0000 7f6ee7fff700 1 --2- 172.21.15.38:0/1541555294 >> [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940] conn(0x7f6ee806ef70 0x7f6ee8071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:41.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.579+0000 7f6ee7fff700 1 -- 172.21.15.38:0/1541555294 --> [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ee8071a30 con 0x7f6ee806ef70 2024-04-22T21:06:41.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.579+0000 7f6ee7fff700 1 -- 172.21.15.38:0/1541555294 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ef4087910 con 0x7f6f001037c0 2024-04-22T21:06:41.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.579+0000 7f6efffff700 1 --2- 172.21.15.38:0/1541555294 >> [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940] conn(0x7f6ee806ef70 0x7f6ee8071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:41.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.579+0000 7f6eff7fe700 1 --2- 172.21.15.38:0/1541555294 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6ee80692a0 0x7f6ee806b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0004f90 tx=0x7f6ef0009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:41.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.579+0000 7f6efffff700 1 --2- 172.21.15.38:0/1541555294 >> [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940] conn(0x7f6ee806ef70 0x7f6ee8071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:41.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.580+0000 7f6ee7fff700 1 -- 172.21.15.38:0/1541555294 <== osd.1 v2:172.21.15.38:6810/1336148940 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f6ee8071a30 con 0x7f6ee806ef70 2024-04-22T21:06:41.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.595+0000 7f6f05fc1700 1 -- 172.21.15.38:0/1541555294 --> [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6f00070580 con 0x7f6ee806ef70 2024-04-22T21:06:41.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.595+0000 7f6ee7fff700 1 -- 172.21.15.38:0/1541555294 <== osd.1 v2:172.21.15.38:6810/1336148940 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6f00070580 con 0x7f6ee806ef70 2024-04-22T21:06:41.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.595+0000 7f6f05fc1700 1 -- 172.21.15.38:0/1541555294 >> [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940] conn(0x7f6ee806ef70 msgr2=0x7f6ee8071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:41.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.595+0000 7f6f05fc1700 1 --2- 172.21.15.38:0/1541555294 >> [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940] conn(0x7f6ee806ef70 0x7f6ee8071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.596+0000 7f6f05fc1700 1 -- 172.21.15.38:0/1541555294 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6ee80692a0 msgr2=0x7f6ee806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:41.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.596+0000 7f6f05fc1700 1 --2- 172.21.15.38:0/1541555294 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6ee80692a0 0x7f6ee806b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0004f90 tx=0x7f6ef0009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:41.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.596+0000 7f6f05fc1700 1 -- 172.21.15.38:0/1541555294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f001037c0 msgr2=0x7f6f001160c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:41.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.596+0000 7f6f05fc1700 1 --2- 172.21.15.38:0/1541555294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f001037c0 0x7f6f001160c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400b790 tx=0x7f6ef4009220 comp rx=0 tx=0).stop 2024-04-22T21:06:41.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.596+0000 7f6f05fc1700 1 -- 172.21.15.38:0/1541555294 shutdown_connections 2024-04-22T21:06:41.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.596+0000 7f6f05fc1700 1 --2- 172.21.15.38:0/1541555294 >> [v2:172.21.15.38:6810/1336148940,v1:172.21.15.38:6811/1336148940] conn(0x7f6ee806ef70 0x7f6ee8071380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.596+0000 7f6f05fc1700 1 --2- 172.21.15.38:0/1541555294 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f6ee80692a0 0x7f6ee806b750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.596+0000 7f6f05fc1700 1 --2- 172.21.15.38:0/1541555294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f00102e20 0x7f6f00115b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.596+0000 7f6f05fc1700 1 --2- 172.21.15.38:0/1541555294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f001037c0 0x7f6f001160c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:41.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.596+0000 7f6f05fc1700 1 -- 172.21.15.38:0/1541555294 >> 172.21.15.38:0/1541555294 conn(0x7f6f000f89b0 msgr2=0x7f6f00071a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:41.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.596+0000 7f6f05fc1700 1 -- 172.21.15.38:0/1541555294 shutdown_connections 2024-04-22T21:06:41.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:41.596+0000 7f6f05fc1700 1 -- 172.21.15.38:0/1541555294 wait complete. 2024-04-22T21:06:41.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:41 smithi098 ceph-mon[34053]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:42.234 INFO:teuthology.orchestra.run.smithi038.stdout:60129542173 2024-04-22T21:06:42.235 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-22T21:06:42.760 INFO:teuthology.orchestra.run.smithi038.stdout:38654705700 2024-04-22T21:06:42.760 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-22T21:06:43.162 INFO:teuthology.orchestra.run.smithi038.stdout:81604378649 2024-04-22T21:06:43.163 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-22T21:06:43.244 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:43.244 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:43.303 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:43.303 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:43.393 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:43 smithi038 ceph-mon[26374]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:43.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:43 smithi098 ceph-mon[34053]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:45.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:45 smithi038 ceph-mon[26374]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:45.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:45 smithi098 ceph-mon[34053]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:46.542 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:47.535 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:47 smithi038 ceph-mon[26374]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:47.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:47 smithi098 ceph-mon[34053]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:48.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.811+0000 7f774b8a7700 1 -- 172.21.15.38:0/2483734234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7744102e00 msgr2=0x7f77441031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:48.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.811+0000 7f774b8a7700 1 --2- 172.21.15.38:0/2483734234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7744102e00 0x7f77441031d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7740004940 tx=0x7f7740009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:48.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.811+0000 7f774b8a7700 1 -- 172.21.15.38:0/2483734234 shutdown_connections 2024-04-22T21:06:48.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.811+0000 7f774b8a7700 1 --2- 172.21.15.38:0/2483734234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77441037a0 0x7f77440fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:48.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.811+0000 7f774b8a7700 1 --2- 172.21.15.38:0/2483734234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7744102e00 0x7f77441031d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:48.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.811+0000 7f774b8a7700 1 -- 172.21.15.38:0/2483734234 >> 172.21.15.38:0/2483734234 conn(0x7f77440f8950 msgr2=0x7f77440fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:48.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.811+0000 7f774b8a7700 1 -- 172.21.15.38:0/2483734234 shutdown_connections 2024-04-22T21:06:48.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.811+0000 7f774b8a7700 1 -- 172.21.15.38:0/2483734234 wait complete. 2024-04-22T21:06:48.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.811+0000 7f774b8a7700 1 Processor -- start 2024-04-22T21:06:48.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.811+0000 7f774b8a7700 1 -- start start 2024-04-22T21:06:48.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.812+0000 7f774b8a7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7744102e00 0x7f7744113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:48.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.812+0000 7f774b8a7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77441037a0 0x7f7744113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.812+0000 7f774b8a7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7744114510 con 0x7f7744102e00 2024-04-22T21:06:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.812+0000 7f774b8a7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f774410db10 con 0x7f77441037a0 2024-04-22T21:06:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.812+0000 7f7749643700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7744102e00 0x7f7744113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.812+0000 7f7749643700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7744102e00 0x7f7744113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48148/0 (socket says 172.21.15.38:48148) 2024-04-22T21:06:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.812+0000 7f7749643700 1 -- 172.21.15.38:0/1180964698 learned_addr learned my addr 172.21.15.38:0/1180964698 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.812+0000 7f7748e42700 1 --2- 172.21.15.38:0/1180964698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77441037a0 0x7f7744113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.812+0000 7f7749643700 1 -- 172.21.15.38:0/1180964698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77441037a0 msgr2=0x7f7744113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.812+0000 7f7749643700 1 --2- 172.21.15.38:0/1180964698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77441037a0 0x7f7744113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.812+0000 7f7749643700 1 -- 172.21.15.38:0/1180964698 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7740005040 con 0x7f7744102e00 2024-04-22T21:06:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.812+0000 7f7749643700 1 --2- 172.21.15.38:0/1180964698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7744102e00 0x7f7744113980 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f774000b3b0 tx=0x7f7740009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:48.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.812+0000 7f7739ffb700 1 -- 172.21.15.38:0/1180964698 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f774000e040 con 0x7f7744102e00 2024-04-22T21:06:48.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.812+0000 7f774b8a7700 1 -- 172.21.15.38:0/1180964698 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f774410dd90 con 0x7f7744102e00 2024-04-22T21:06:48.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.812+0000 7f7739ffb700 1 -- 172.21.15.38:0/1180964698 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f774000bbc0 con 0x7f7744102e00 2024-04-22T21:06:48.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.812+0000 7f7739ffb700 1 -- 172.21.15.38:0/1180964698 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f774001c3f0 con 0x7f7744102e00 2024-04-22T21:06:48.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.813+0000 7f774b8a7700 1 -- 172.21.15.38:0/1180964698 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f774410e350 con 0x7f7744102e00 2024-04-22T21:06:48.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.814+0000 7f774b8a7700 1 -- 172.21.15.38:0/1180964698 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7744100110 con 0x7f7744102e00 2024-04-22T21:06:48.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.814+0000 7f7739ffb700 1 -- 172.21.15.38:0/1180964698 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f774000bd30 con 0x7f7744102e00 2024-04-22T21:06:48.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.815+0000 7f7739ffb700 1 --2- 172.21.15.38:0/1180964698 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f77300692a0 0x7f773006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:48.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.815+0000 7f7739ffb700 1 -- 172.21.15.38:0/1180964698 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7740088100 con 0x7f7744102e00 2024-04-22T21:06:48.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.815+0000 7f7748e42700 1 --2- 172.21.15.38:0/1180964698 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f77300692a0 0x7f773006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:48.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.815+0000 7f7748e42700 1 --2- 172.21.15.38:0/1180964698 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f77300692a0 0x7f773006b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f774410ecd0 tx=0x7f7734009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:48.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:48.819+0000 7f7739ffb700 1 -- 172.21.15.38:0/1180964698 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f7740058230 con 0x7f7744102e00 2024-04-22T21:06:49.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.021+0000 7f774b8a7700 1 -- 172.21.15.38:0/1180964698 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f774410efd0 con 0x7f7744102e00 2024-04-22T21:06:49.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.022+0000 7f7739ffb700 1 -- 172.21.15.38:0/1180964698 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7740057dc0 con 0x7f7744102e00 2024-04-22T21:06:49.018 INFO:teuthology.orchestra.run.smithi038.stdout:150323855375 2024-04-22T21:06:49.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.024+0000 7f774b8a7700 1 -- 172.21.15.38:0/1180964698 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f77300692a0 msgr2=0x7f773006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:49.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.024+0000 7f774b8a7700 1 --2- 172.21.15.38:0/1180964698 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f77300692a0 0x7f773006b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f774410ecd0 tx=0x7f7734009ee0 comp rx=0 tx=0).stop 2024-04-22T21:06:49.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.024+0000 7f774b8a7700 1 -- 172.21.15.38:0/1180964698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7744102e00 msgr2=0x7f7744113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:49.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.024+0000 7f774b8a7700 1 --2- 172.21.15.38:0/1180964698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7744102e00 0x7f7744113980 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f774000b3b0 tx=0x7f7740009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:49.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.024+0000 7f774b8a7700 1 -- 172.21.15.38:0/1180964698 shutdown_connections 2024-04-22T21:06:49.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.024+0000 7f774b8a7700 1 --2- 172.21.15.38:0/1180964698 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f77300692a0 0x7f773006b750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:49.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.024+0000 7f774b8a7700 1 --2- 172.21.15.38:0/1180964698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7744102e00 0x7f7744113980 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:49.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.024+0000 7f774b8a7700 1 --2- 172.21.15.38:0/1180964698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77441037a0 0x7f7744113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:49.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.024+0000 7f774b8a7700 1 -- 172.21.15.38:0/1180964698 >> 172.21.15.38:0/1180964698 conn(0x7f77440f8950 msgr2=0x7f77441073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:49.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.024+0000 7f774b8a7700 1 -- 172.21.15.38:0/1180964698 shutdown_connections 2024-04-22T21:06:49.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.025+0000 7f774b8a7700 1 -- 172.21.15.38:0/1180964698 wait complete. 2024-04-22T21:06:49.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:49 smithi098 ceph-mon[34053]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:49.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:49 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/1180964698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T21:06:49.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:49 smithi038 ceph-mon[26374]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:49.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:49 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1180964698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T21:06:49.729 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855372 got 150323855375 for osd.4 2024-04-22T21:06:49.729 DEBUG:teuthology.parallel:result is None 2024-04-22T21:06:50.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.079+0000 7fcb4c66c700 1 -- 172.21.15.38:0/1164244435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb441026b0 msgr2=0x7fcb440fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:50.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.079+0000 7fcb4c66c700 1 --2- 172.21.15.38:0/1164244435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb441026b0 0x7fcb440fbd20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcb34004940 tx=0x7fcb34009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:50.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.080+0000 7fcb4c66c700 1 -- 172.21.15.38:0/1164244435 shutdown_connections 2024-04-22T21:06:50.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.080+0000 7fcb4c66c700 1 --2- 172.21.15.38:0/1164244435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb441026b0 0x7fcb440fbd20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:50.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.080+0000 7fcb4c66c700 1 --2- 172.21.15.38:0/1164244435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb44101d10 0x7fcb441020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:50.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.080+0000 7fcb4c66c700 1 -- 172.21.15.38:0/1164244435 >> 172.21.15.38:0/1164244435 conn(0x7fcb44072bc0 msgr2=0x7fcb44072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:50.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.080+0000 7fcb4c66c700 1 -- 172.21.15.38:0/1164244435 shutdown_connections 2024-04-22T21:06:50.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.080+0000 7fcb4c66c700 1 -- 172.21.15.38:0/1164244435 wait complete. 2024-04-22T21:06:50.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.080+0000 7fcb4c66c700 1 Processor -- start 2024-04-22T21:06:50.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.080+0000 7fcb4c66c700 1 -- start start 2024-04-22T21:06:50.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.080+0000 7fcb4c66c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb44101d10 0x7fcb44113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:50.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.080+0000 7fcb4c66c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb441026b0 0x7fcb441140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:50.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.080+0000 7fcb4c66c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb44114720 con 0x7fcb44101d10 2024-04-22T21:06:50.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.080+0000 7fcb4c66c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4410db10 con 0x7fcb441026b0 2024-04-22T21:06:50.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.081+0000 7fcb4a408700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb44101d10 0x7fcb44113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:50.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.081+0000 7fcb4a408700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb44101d10 0x7fcb44113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48186/0 (socket says 172.21.15.38:48186) 2024-04-22T21:06:50.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.081+0000 7fcb4a408700 1 -- 172.21.15.38:0/3530297351 learned_addr learned my addr 172.21.15.38:0/3530297351 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:50.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.081+0000 7fcb49c07700 1 --2- 172.21.15.38:0/3530297351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb441026b0 0x7fcb441140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:50.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.081+0000 7fcb4a408700 1 -- 172.21.15.38:0/3530297351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb441026b0 msgr2=0x7fcb441140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:50.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.081+0000 7fcb4a408700 1 --2- 172.21.15.38:0/3530297351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb441026b0 0x7fcb441140d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:50.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.081+0000 7fcb4a408700 1 -- 172.21.15.38:0/3530297351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb34005040 con 0x7fcb44101d10 2024-04-22T21:06:50.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.081+0000 7fcb4a408700 1 --2- 172.21.15.38:0/3530297351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb44101d10 0x7fcb44113b90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcb40002a50 tx=0x7fcb400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:50.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.081+0000 7fcb3affd700 1 -- 172.21.15.38:0/3530297351 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb40007cc0 con 0x7fcb44101d10 2024-04-22T21:06:50.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.081+0000 7fcb4c66c700 1 -- 172.21.15.38:0/3530297351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4410ddf0 con 0x7fcb44101d10 2024-04-22T21:06:50.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.081+0000 7fcb4c66c700 1 -- 172.21.15.38:0/3530297351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4410e380 con 0x7fcb44101d10 2024-04-22T21:06:50.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.081+0000 7fcb3affd700 1 -- 172.21.15.38:0/3530297351 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb4000f040 con 0x7fcb44101d10 2024-04-22T21:06:50.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.081+0000 7fcb3affd700 1 -- 172.21.15.38:0/3530297351 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb40009640 con 0x7fcb44101d10 2024-04-22T21:06:50.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.082+0000 7fcb4c66c700 1 -- 172.21.15.38:0/3530297351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb440fd460 con 0x7fcb44101d10 2024-04-22T21:06:50.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.083+0000 7fcb3affd700 1 -- 172.21.15.38:0/3530297351 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fcb400097a0 con 0x7fcb44101d10 2024-04-22T21:06:50.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.083+0000 7fcb3affd700 1 --2- 172.21.15.38:0/3530297351 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fcb300692b0 0x7fcb3006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:50.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.083+0000 7fcb3affd700 1 -- 172.21.15.38:0/3530297351 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcb40086b80 con 0x7fcb44101d10 2024-04-22T21:06:50.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.083+0000 7fcb49c07700 1 --2- 172.21.15.38:0/3530297351 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fcb300692b0 0x7fcb3006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:50.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.084+0000 7fcb49c07700 1 --2- 172.21.15.38:0/3530297351 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fcb300692b0 0x7fcb3006b760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcb4410edf0 tx=0x7fcb34009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:50.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.088+0000 7fcb3affd700 1 -- 172.21.15.38:0/3530297351 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fcb40056c80 con 0x7fcb44101d10 2024-04-22T21:06:50.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.982+0000 7f098fe85700 1 -- 172.21.15.38:0/1923953976 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09880fdcc0 msgr2=0x7f09880fe090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:50.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.982+0000 7f098fe85700 1 --2- 172.21.15.38:0/1923953976 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09880fdcc0 0x7f09880fe090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0984004940 tx=0x7f0984009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:50.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.982+0000 7f098fe85700 1 -- 172.21.15.38:0/1923953976 shutdown_connections 2024-04-22T21:06:50.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.982+0000 7f098fe85700 1 --2- 172.21.15.38:0/1923953976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09880fb4a0 0x7f09880fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:50.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.982+0000 7f098fe85700 1 --2- 172.21.15.38:0/1923953976 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09880fdcc0 0x7f09880fe090 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:50.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.982+0000 7f098fe85700 1 -- 172.21.15.38:0/1923953976 >> 172.21.15.38:0/1923953976 conn(0x7f09880714c0 msgr2=0x7f09880718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:50.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.982+0000 7f098fe85700 1 -- 172.21.15.38:0/1923953976 shutdown_connections 2024-04-22T21:06:50.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.982+0000 7f098fe85700 1 -- 172.21.15.38:0/1923953976 wait complete. 2024-04-22T21:06:50.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.983+0000 7f098fe85700 1 Processor -- start 2024-04-22T21:06:50.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.983+0000 7f098fe85700 1 -- start start 2024-04-22T21:06:50.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.983+0000 7f098fe85700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09880fb4a0 0x7f0988115db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:50.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.983+0000 7f098fe85700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09880fdcc0 0x7f09881162f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:50.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.983+0000 7f098fe85700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09881169d0 con 0x7f09880fb4a0 2024-04-22T21:06:50.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.983+0000 7f098fe85700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098810fd10 con 0x7f09880fdcc0 2024-04-22T21:06:50.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.983+0000 7f098dc21700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09880fb4a0 0x7f0988115db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:50.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.983+0000 7f098dc21700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09880fb4a0 0x7f0988115db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48166/0 (socket says 172.21.15.38:48166) 2024-04-22T21:06:50.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.983+0000 7f098dc21700 1 -- 172.21.15.38:0/838084047 learned_addr learned my addr 172.21.15.38:0/838084047 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:50.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.983+0000 7f098d420700 1 --2- 172.21.15.38:0/838084047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09880fdcc0 0x7f09881162f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:50.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.984+0000 7f098dc21700 1 -- 172.21.15.38:0/838084047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09880fdcc0 msgr2=0x7f09881162f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:50.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.984+0000 7f098dc21700 1 --2- 172.21.15.38:0/838084047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09880fdcc0 0x7f09881162f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:50.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.984+0000 7f098dc21700 1 -- 172.21.15.38:0/838084047 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0984005040 con 0x7f09880fb4a0 2024-04-22T21:06:50.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.984+0000 7f098d420700 1 --2- 172.21.15.38:0/838084047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09880fdcc0 0x7f09881162f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T21:06:50.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.984+0000 7f098dc21700 1 --2- 172.21.15.38:0/838084047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09880fb4a0 0x7f0988115db0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0984009150 tx=0x7f0984009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:50.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.984+0000 7f097e7fc700 1 -- 172.21.15.38:0/838084047 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f098400e040 con 0x7f09880fb4a0 2024-04-22T21:06:50.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.984+0000 7f098fe85700 1 -- 172.21.15.38:0/838084047 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098810ff30 con 0x7f09880fb4a0 2024-04-22T21:06:50.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.984+0000 7f097e7fc700 1 -- 172.21.15.38:0/838084047 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f098400bbc0 con 0x7f09880fb4a0 2024-04-22T21:06:50.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.984+0000 7f097e7fc700 1 -- 172.21.15.38:0/838084047 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f098401c630 con 0x7f09880fb4a0 2024-04-22T21:06:50.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.984+0000 7f098fe85700 1 -- 172.21.15.38:0/838084047 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09881104f0 con 0x7f09880fb4a0 2024-04-22T21:06:50.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.986+0000 7f097e7fc700 1 -- 172.21.15.38:0/838084047 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f098400bd30 con 0x7f09880fb4a0 2024-04-22T21:06:50.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.986+0000 7f098fe85700 1 -- 172.21.15.38:0/838084047 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09880fc4b0 con 0x7f09880fb4a0 2024-04-22T21:06:50.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.986+0000 7f097e7fc700 1 --2- 172.21.15.38:0/838084047 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f09740692b0 0x7f097406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:50.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.986+0000 7f097e7fc700 1 -- 172.21.15.38:0/838084047 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0984088170 con 0x7f09880fb4a0 2024-04-22T21:06:50.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.986+0000 7f098d420700 1 --2- 172.21.15.38:0/838084047 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f09740692b0 0x7f097406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:50.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.987+0000 7f098d420700 1 --2- 172.21.15.38:0/838084047 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f09740692b0 0x7f097406b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0988111650 tx=0x7f0978008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:50.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:49.992+0000 7f097e7fc700 1 -- 172.21.15.38:0/838084047 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f0984058270 con 0x7f09880fb4a0 2024-04-22T21:06:50.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.191+0000 7f098fe85700 1 -- 172.21.15.38:0/838084047 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f098805f5f0 con 0x7f09880fb4a0 2024-04-22T21:06:50.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.192+0000 7f097e7fc700 1 -- 172.21.15.38:0/838084047 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0984057e00 con 0x7f09880fb4a0 2024-04-22T21:06:50.188 INFO:teuthology.orchestra.run.smithi038.stdout:115964117012 2024-04-22T21:06:50.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.193+0000 7f098fe85700 1 -- 172.21.15.38:0/838084047 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f09740692b0 msgr2=0x7f097406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:50.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.193+0000 7f098fe85700 1 --2- 172.21.15.38:0/838084047 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f09740692b0 0x7f097406b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0988111650 tx=0x7f0978008040 comp rx=0 tx=0).stop 2024-04-22T21:06:50.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.194+0000 7f098fe85700 1 -- 172.21.15.38:0/838084047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09880fb4a0 msgr2=0x7f0988115db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:50.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.194+0000 7f098fe85700 1 --2- 172.21.15.38:0/838084047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09880fb4a0 0x7f0988115db0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0984009150 tx=0x7f0984009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:50.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.194+0000 7f098fe85700 1 -- 172.21.15.38:0/838084047 shutdown_connections 2024-04-22T21:06:50.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.194+0000 7f098fe85700 1 --2- 172.21.15.38:0/838084047 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f09740692b0 0x7f097406b760 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:50.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.194+0000 7f098fe85700 1 --2- 172.21.15.38:0/838084047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09880fb4a0 0x7f0988115db0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:50.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.194+0000 7f098fe85700 1 --2- 172.21.15.38:0/838084047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09880fdcc0 0x7f09881162f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:50.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.194+0000 7f098fe85700 1 -- 172.21.15.38:0/838084047 >> 172.21.15.38:0/838084047 conn(0x7f09880714c0 msgr2=0x7f09880f9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:50.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.194+0000 7f098fe85700 1 -- 172.21.15.38:0/838084047 shutdown_connections 2024-04-22T21:06:50.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.194+0000 7f098fe85700 1 -- 172.21.15.38:0/838084047 wait complete. 2024-04-22T21:06:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.290+0000 7fcb4c66c700 1 -- 172.21.15.38:0/3530297351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fcb44108e20 con 0x7fcb44101d10 2024-04-22T21:06:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.291+0000 7fcb3affd700 1 -- 172.21.15.38:0/3530297351 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcb40056810 con 0x7fcb44101d10 2024-04-22T21:06:50.287 INFO:teuthology.orchestra.run.smithi038.stdout:60129542175 2024-04-22T21:06:50.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.293+0000 7fcb4c66c700 1 -- 172.21.15.38:0/3530297351 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fcb300692b0 msgr2=0x7fcb3006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:50.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.293+0000 7fcb4c66c700 1 --2- 172.21.15.38:0/3530297351 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fcb300692b0 0x7fcb3006b760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcb4410edf0 tx=0x7fcb34009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:50.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.293+0000 7fcb4c66c700 1 -- 172.21.15.38:0/3530297351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb44101d10 msgr2=0x7fcb44113b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:50.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.293+0000 7fcb4c66c700 1 --2- 172.21.15.38:0/3530297351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb44101d10 0x7fcb44113b90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcb40002a50 tx=0x7fcb400046b0 comp rx=0 tx=0).stop 2024-04-22T21:06:50.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.293+0000 7fcb4c66c700 1 -- 172.21.15.38:0/3530297351 shutdown_connections 2024-04-22T21:06:50.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.293+0000 7fcb4c66c700 1 --2- 172.21.15.38:0/3530297351 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fcb300692b0 0x7fcb3006b760 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:50.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.293+0000 7fcb4c66c700 1 --2- 172.21.15.38:0/3530297351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb44101d10 0x7fcb44113b90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:50.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.293+0000 7fcb4c66c700 1 --2- 172.21.15.38:0/3530297351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb441026b0 0x7fcb441140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:50.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.293+0000 7fcb4c66c700 1 -- 172.21.15.38:0/3530297351 >> 172.21.15.38:0/3530297351 conn(0x7fcb44072bc0 msgr2=0x7fcb440655d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:50.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.293+0000 7fcb4c66c700 1 -- 172.21.15.38:0/3530297351 shutdown_connections 2024-04-22T21:06:50.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:50.293+0000 7fcb4c66c700 1 -- 172.21.15.38:0/3530297351 wait complete. 2024-04-22T21:06:50.582 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:50 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/838084047' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T21:06:50.582 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:50 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3530297351' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T21:06:50.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:50 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/838084047' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T21:06:50.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:50 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/3530297351' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T21:06:51.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.208+0000 7f4540325700 1 -- 172.21.15.38:0/3794041143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4538102e10 msgr2=0x7f45381031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:51.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.208+0000 7f4540325700 1 --2- 172.21.15.38:0/3794041143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4538102e10 0x7f45381031e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4530004940 tx=0x7f4530009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:51.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.208+0000 7f4540325700 1 -- 172.21.15.38:0/3794041143 shutdown_connections 2024-04-22T21:06:51.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.208+0000 7f4540325700 1 --2- 172.21.15.38:0/3794041143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45381037b0 0x7f45380fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:51.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.208+0000 7f4540325700 1 --2- 172.21.15.38:0/3794041143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4538102e10 0x7f45381031e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:51.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.208+0000 7f4540325700 1 -- 172.21.15.38:0/3794041143 >> 172.21.15.38:0/3794041143 conn(0x7f45380f89a0 msgr2=0x7f45380fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:51.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.208+0000 7f4540325700 1 -- 172.21.15.38:0/3794041143 shutdown_connections 2024-04-22T21:06:51.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.208+0000 7f4540325700 1 -- 172.21.15.38:0/3794041143 wait complete. 2024-04-22T21:06:51.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.209+0000 7f4540325700 1 Processor -- start 2024-04-22T21:06:51.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.209+0000 7f4540325700 1 -- start start 2024-04-22T21:06:51.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.209+0000 7f4540325700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4538102e10 0x7f4538113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:51.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.209+0000 7f4540325700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45381037b0 0x7f45381140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:51.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.209+0000 7f4540325700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45381147d0 con 0x7f4538102e10 2024-04-22T21:06:51.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.209+0000 7f4540325700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453810db30 con 0x7f45381037b0 2024-04-22T21:06:51.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.209+0000 7f453e0c1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4538102e10 0x7f4538113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:51.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.209+0000 7f453e0c1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4538102e10 0x7f4538113bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48196/0 (socket says 172.21.15.38:48196) 2024-04-22T21:06:51.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.209+0000 7f453e0c1700 1 -- 172.21.15.38:0/548265130 learned_addr learned my addr 172.21.15.38:0/548265130 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:51.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.209+0000 7f453e0c1700 1 -- 172.21.15.38:0/548265130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45381037b0 msgr2=0x7f45381140f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:51.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.209+0000 7f453e0c1700 1 --2- 172.21.15.38:0/548265130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45381037b0 0x7f45381140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:51.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.209+0000 7f453e0c1700 1 -- 172.21.15.38:0/548265130 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4530005040 con 0x7f4538102e10 2024-04-22T21:06:51.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.210+0000 7f453e0c1700 1 --2- 172.21.15.38:0/548265130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4538102e10 0x7f4538113bb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f453000b3b0 tx=0x7f4530009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:51.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.210+0000 7f4536ffd700 1 -- 172.21.15.38:0/548265130 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f453000e040 con 0x7f4538102e10 2024-04-22T21:06:51.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.210+0000 7f4536ffd700 1 -- 172.21.15.38:0/548265130 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f453000bbc0 con 0x7f4538102e10 2024-04-22T21:06:51.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.210+0000 7f4540325700 1 -- 172.21.15.38:0/548265130 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f453810ddb0 con 0x7f4538102e10 2024-04-22T21:06:51.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.210+0000 7f4536ffd700 1 -- 172.21.15.38:0/548265130 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f453001c630 con 0x7f4538102e10 2024-04-22T21:06:51.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.210+0000 7f4540325700 1 -- 172.21.15.38:0/548265130 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f453810e370 con 0x7f4538102e10 2024-04-22T21:06:51.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.211+0000 7f4540325700 1 -- 172.21.15.38:0/548265130 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45380fe560 con 0x7f4538102e10 2024-04-22T21:06:51.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.211+0000 7f4536ffd700 1 -- 172.21.15.38:0/548265130 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f453000bd30 con 0x7f4538102e10 2024-04-22T21:06:51.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.212+0000 7f4536ffd700 1 --2- 172.21.15.38:0/548265130 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f45200692a0 0x7f452006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:51.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.212+0000 7f4536ffd700 1 -- 172.21.15.38:0/548265130 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f45300880e0 con 0x7f4538102e10 2024-04-22T21:06:51.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.212+0000 7f453d8c0700 1 --2- 172.21.15.38:0/548265130 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f45200692a0 0x7f452006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:51.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.212+0000 7f453d8c0700 1 --2- 172.21.15.38:0/548265130 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f45200692a0 0x7f452006b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f452c006fd0 tx=0x7f452c006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:51.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.216+0000 7f4536ffd700 1 -- 172.21.15.38:0/548265130 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f45300581e0 con 0x7f4538102e10 2024-04-22T21:06:51.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.421+0000 7f4540325700 1 -- 172.21.15.38:0/548265130 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4538109f80 con 0x7f4538102e10 2024-04-22T21:06:51.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.421+0000 7f4536ffd700 1 -- 172.21.15.38:0/548265130 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4530057d70 con 0x7f4538102e10 2024-04-22T21:06:51.417 INFO:teuthology.orchestra.run.smithi038.stdout:38654705702 2024-04-22T21:06:51.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.423+0000 7f4540325700 1 -- 172.21.15.38:0/548265130 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f45200692a0 msgr2=0x7f452006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:51.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.423+0000 7f4540325700 1 --2- 172.21.15.38:0/548265130 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f45200692a0 0x7f452006b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f452c006fd0 tx=0x7f452c006db0 comp rx=0 tx=0).stop 2024-04-22T21:06:51.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.423+0000 7f4540325700 1 -- 172.21.15.38:0/548265130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4538102e10 msgr2=0x7f4538113bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:51.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.423+0000 7f4540325700 1 --2- 172.21.15.38:0/548265130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4538102e10 0x7f4538113bb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f453000b3b0 tx=0x7f4530009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:51.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.423+0000 7f4540325700 1 -- 172.21.15.38:0/548265130 shutdown_connections 2024-04-22T21:06:51.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.423+0000 7f4540325700 1 --2- 172.21.15.38:0/548265130 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f45200692a0 0x7f452006b750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:51.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.423+0000 7f4540325700 1 --2- 172.21.15.38:0/548265130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4538102e10 0x7f4538113bb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:51.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.423+0000 7f4540325700 1 --2- 172.21.15.38:0/548265130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45381037b0 0x7f45381140f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:51.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.423+0000 7f4540325700 1 -- 172.21.15.38:0/548265130 >> 172.21.15.38:0/548265130 conn(0x7f45380f89a0 msgr2=0x7f45380f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:51.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.423+0000 7f4540325700 1 -- 172.21.15.38:0/548265130 shutdown_connections 2024-04-22T21:06:51.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.423+0000 7f4540325700 1 -- 172.21.15.38:0/548265130 wait complete. 2024-04-22T21:06:51.430 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:51 smithi038 ceph-mon[26374]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:51.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:51 smithi098 ceph-mon[34053]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:51.876 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117010 got 115964117012 for osd.3 2024-04-22T21:06:51.876 DEBUG:teuthology.parallel:result is None 2024-04-22T21:06:52.570 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:52 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/548265130' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T21:06:52.594 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705702 for osd.0 2024-04-22T21:06:52.594 DEBUG:teuthology.parallel:result is None 2024-04-22T21:06:52.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:52 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/548265130' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T21:06:52.888 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542173 got 60129542175 for osd.1 2024-04-22T21:06:52.889 DEBUG:teuthology.parallel:result is None 2024-04-22T21:06:52.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.575+0000 7f929dea9700 1 -- 172.21.15.38:0/3385419559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9298101d20 msgr2=0x7f92981020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:52.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.575+0000 7f929dea9700 1 --2- 172.21.15.38:0/3385419559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9298101d20 0x7f92981020f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9288004940 tx=0x7f9288009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:52.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.576+0000 7f929dea9700 1 -- 172.21.15.38:0/3385419559 shutdown_connections 2024-04-22T21:06:52.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.576+0000 7f929dea9700 1 --2- 172.21.15.38:0/3385419559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92981026c0 0x7f92980fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:52.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.576+0000 7f929dea9700 1 --2- 172.21.15.38:0/3385419559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9298101d20 0x7f92981020f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:52.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.576+0000 7f929dea9700 1 -- 172.21.15.38:0/3385419559 >> 172.21.15.38:0/3385419559 conn(0x7f92980f7870 msgr2=0x7f92980f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:52.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.576+0000 7f929dea9700 1 -- 172.21.15.38:0/3385419559 shutdown_connections 2024-04-22T21:06:52.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.576+0000 7f929dea9700 1 -- 172.21.15.38:0/3385419559 wait complete. 2024-04-22T21:06:52.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.576+0000 7f929dea9700 1 Processor -- start 2024-04-22T21:06:52.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.576+0000 7f929dea9700 1 -- start start 2024-04-22T21:06:52.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.577+0000 7f929dea9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9298101d20 0x7f9298102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:52.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.577+0000 7f929dea9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92981026c0 0x7f92981030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:52.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.577+0000 7f929dea9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9298104ad0 con 0x7f92981026c0 2024-04-22T21:06:52.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.577+0000 7f929dea9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9298103600 con 0x7f9298101d20 2024-04-22T21:06:52.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.577+0000 7f9296ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92981026c0 0x7f92981030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.577+0000 7f9296ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92981026c0 0x7f92981030c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48214/0 (socket says 172.21.15.38:48214) 2024-04-22T21:06:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.577+0000 7f9296ffd700 1 -- 172.21.15.38:0/148422509 learned_addr learned my addr 172.21.15.38:0/148422509 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.577+0000 7f92977fe700 1 --2- 172.21.15.38:0/148422509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9298101d20 0x7f9298102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.577+0000 7f9296ffd700 1 -- 172.21.15.38:0/148422509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9298101d20 msgr2=0x7f9298102b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.577+0000 7f9296ffd700 1 --2- 172.21.15.38:0/148422509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9298101d20 0x7f9298102b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.577+0000 7f9296ffd700 1 -- 172.21.15.38:0/148422509 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9288005040 con 0x7f92981026c0 2024-04-22T21:06:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.577+0000 7f9296ffd700 1 --2- 172.21.15.38:0/148422509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92981026c0 0x7f92981030c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f928c002a50 tx=0x7f928c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.577+0000 7f927ffff700 1 -- 172.21.15.38:0/148422509 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f928c007cc0 con 0x7f92981026c0 2024-04-22T21:06:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.578+0000 7f929dea9700 1 -- 172.21.15.38:0/148422509 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92981038e0 con 0x7f92981026c0 2024-04-22T21:06:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.578+0000 7f929dea9700 1 -- 172.21.15.38:0/148422509 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929811a180 con 0x7f92981026c0 2024-04-22T21:06:52.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.578+0000 7f927ffff700 1 -- 172.21.15.38:0/148422509 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f928c00f040 con 0x7f92981026c0 2024-04-22T21:06:52.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.578+0000 7f927ffff700 1 -- 172.21.15.38:0/148422509 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f928c00d9c0 con 0x7f92981026c0 2024-04-22T21:06:52.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.578+0000 7f929dea9700 1 -- 172.21.15.38:0/148422509 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9284000fd0 con 0x7f92981026c0 2024-04-22T21:06:52.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.579+0000 7f927ffff700 1 -- 172.21.15.38:0/148422509 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f928c00db40 con 0x7f92981026c0 2024-04-22T21:06:52.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.580+0000 7f927ffff700 1 --2- 172.21.15.38:0/148422509 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f92800692a0 0x7f928006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:52.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.580+0000 7f927ffff700 1 -- 172.21.15.38:0/148422509 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f928c0875d0 con 0x7f92981026c0 2024-04-22T21:06:52.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.580+0000 7f92977fe700 1 --2- 172.21.15.38:0/148422509 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f92800692a0 0x7f928006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:52.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.580+0000 7f927ffff700 1 --2- 172.21.15.38:0/148422509 >> [v2:172.21.15.98:6816/2442541230,v1:172.21.15.98:6817/2442541230] conn(0x7f928006ef70 0x7f9280071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:52.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.580+0000 7f927ffff700 1 -- 172.21.15.38:0/148422509 --> [v2:172.21.15.98:6816/2442541230,v1:172.21.15.98:6817/2442541230] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9280071a30 con 0x7f928006ef70 2024-04-22T21:06:52.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.580+0000 7f927ffff700 1 -- 172.21.15.38:0/148422509 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f928c0879b0 con 0x7f92981026c0 2024-04-22T21:06:52.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.580+0000 7f92977fe700 1 --2- 172.21.15.38:0/148422509 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f92800692a0 0x7f928006b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f928800b3b0 tx=0x7f9288009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:52.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.580+0000 7f9297fff700 1 --2- 172.21.15.38:0/148422509 >> [v2:172.21.15.98:6816/2442541230,v1:172.21.15.98:6817/2442541230] conn(0x7f928006ef70 0x7f9280071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:52.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.581+0000 7f9297fff700 1 --2- 172.21.15.38:0/148422509 >> [v2:172.21.15.98:6816/2442541230,v1:172.21.15.98:6817/2442541230] conn(0x7f928006ef70 0x7f9280071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:52.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.581+0000 7f927ffff700 1 -- 172.21.15.38:0/148422509 <== osd.5 v2:172.21.15.98:6816/2442541230 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f9280071a30 con 0x7f928006ef70 2024-04-22T21:06:52.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.596+0000 7f929dea9700 1 -- 172.21.15.38:0/148422509 --> [v2:172.21.15.98:6816/2442541230,v1:172.21.15.98:6817/2442541230] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9284002d30 con 0x7f928006ef70 2024-04-22T21:06:52.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.597+0000 7f927ffff700 1 -- 172.21.15.38:0/148422509 <== osd.5 v2:172.21.15.98:6816/2442541230 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9284002d30 con 0x7f928006ef70 2024-04-22T21:06:52.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.597+0000 7f929dea9700 1 -- 172.21.15.38:0/148422509 >> [v2:172.21.15.98:6816/2442541230,v1:172.21.15.98:6817/2442541230] conn(0x7f928006ef70 msgr2=0x7f9280071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:52.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.597+0000 7f929dea9700 1 --2- 172.21.15.38:0/148422509 >> [v2:172.21.15.98:6816/2442541230,v1:172.21.15.98:6817/2442541230] conn(0x7f928006ef70 0x7f9280071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:52.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.597+0000 7f929dea9700 1 -- 172.21.15.38:0/148422509 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f92800692a0 msgr2=0x7f928006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:52.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.597+0000 7f929dea9700 1 --2- 172.21.15.38:0/148422509 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f92800692a0 0x7f928006b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f928800b3b0 tx=0x7f9288009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:52.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.597+0000 7f929dea9700 1 -- 172.21.15.38:0/148422509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92981026c0 msgr2=0x7f92981030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:52.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.597+0000 7f929dea9700 1 --2- 172.21.15.38:0/148422509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92981026c0 0x7f92981030c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f928c002a50 tx=0x7f928c004600 comp rx=0 tx=0).stop 2024-04-22T21:06:52.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.597+0000 7f929dea9700 1 -- 172.21.15.38:0/148422509 shutdown_connections 2024-04-22T21:06:52.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.597+0000 7f929dea9700 1 --2- 172.21.15.38:0/148422509 >> [v2:172.21.15.98:6816/2442541230,v1:172.21.15.98:6817/2442541230] conn(0x7f928006ef70 0x7f9280071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:52.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.597+0000 7f929dea9700 1 --2- 172.21.15.38:0/148422509 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f92800692a0 0x7f928006b750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:52.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.597+0000 7f929dea9700 1 --2- 172.21.15.38:0/148422509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9298101d20 0x7f9298102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:52.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.597+0000 7f929dea9700 1 --2- 172.21.15.38:0/148422509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92981026c0 0x7f92981030c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:52.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.597+0000 7f929dea9700 1 -- 172.21.15.38:0/148422509 >> 172.21.15.38:0/148422509 conn(0x7f92980f7870 msgr2=0x7f92980f8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:52.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.597+0000 7f929dea9700 1 -- 172.21.15.38:0/148422509 shutdown_connections 2024-04-22T21:06:52.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:51.597+0000 7f929dea9700 1 -- 172.21.15.38:0/148422509 wait complete. 2024-04-22T21:06:53.501 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:53 smithi038 ceph-mon[26374]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:53.501 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:06:53.530 INFO:teuthology.orchestra.run.smithi038.stdout:171798691852 2024-04-22T21:06:53.531 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-22T21:06:53.636 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:53.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:53 smithi098 ceph-mon[34053]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:53.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:06:54.666 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:54 smithi038 ceph-mon[26374]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:54.740 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:54.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:54 smithi098 ceph-mon[34053]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:55.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.286+0000 7fc4e23ce700 1 -- 172.21.15.38:0/3242598604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4dc102730 msgr2=0x7fc4dc0fbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:55.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.286+0000 7fc4e23ce700 1 --2- 172.21.15.38:0/3242598604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4dc102730 0x7fc4dc0fbda0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0004940 tx=0x7fc4d0009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:55.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.286+0000 7fc4e23ce700 1 -- 172.21.15.38:0/3242598604 shutdown_connections 2024-04-22T21:06:55.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.286+0000 7fc4e23ce700 1 --2- 172.21.15.38:0/3242598604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4dc102730 0x7fc4dc0fbda0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:55.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.286+0000 7fc4e23ce700 1 --2- 172.21.15.38:0/3242598604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4dc101d90 0x7fc4dc102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:55.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.286+0000 7fc4e23ce700 1 -- 172.21.15.38:0/3242598604 >> 172.21.15.38:0/3242598604 conn(0x7fc4dc0f7900 msgr2=0x7fc4dc0f9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:55.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.286+0000 7fc4e23ce700 1 -- 172.21.15.38:0/3242598604 shutdown_connections 2024-04-22T21:06:55.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.286+0000 7fc4e23ce700 1 -- 172.21.15.38:0/3242598604 wait complete. 2024-04-22T21:06:55.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.286+0000 7fc4e23ce700 1 Processor -- start 2024-04-22T21:06:55.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.287+0000 7fc4e23ce700 1 -- start start 2024-04-22T21:06:55.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.287+0000 7fc4e23ce700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4dc101d90 0x7fc4dc10eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:55.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.287+0000 7fc4e23ce700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4dc102730 0x7fc4dc10f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:55.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.287+0000 7fc4e23ce700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc067240 con 0x7fc4dc102730 2024-04-22T21:06:55.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.287+0000 7fc4e23ce700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc10f5a0 con 0x7fc4dc101d90 2024-04-22T21:06:55.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.287+0000 7fc4db7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4dc102730 0x7fc4dc10f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:55.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.287+0000 7fc4db7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4dc102730 0x7fc4dc10f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51470/0 (socket says 172.21.15.38:51470) 2024-04-22T21:06:55.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.287+0000 7fc4db7fe700 1 -- 172.21.15.38:0/1889232386 learned_addr learned my addr 172.21.15.38:0/1889232386 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:55.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.287+0000 7fc4dbfff700 1 --2- 172.21.15.38:0/1889232386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4dc101d90 0x7fc4dc10eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:55.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.287+0000 7fc4db7fe700 1 -- 172.21.15.38:0/1889232386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4dc101d90 msgr2=0x7fc4dc10eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:55.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.287+0000 7fc4db7fe700 1 --2- 172.21.15.38:0/1889232386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4dc101d90 0x7fc4dc10eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:55.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.287+0000 7fc4db7fe700 1 -- 172.21.15.38:0/1889232386 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d0005040 con 0x7fc4dc102730 2024-04-22T21:06:55.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.287+0000 7fc4dbfff700 1 --2- 172.21.15.38:0/1889232386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4dc101d90 0x7fc4dc10eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T21:06:55.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.287+0000 7fc4db7fe700 1 --2- 172.21.15.38:0/1889232386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4dc102730 0x7fc4dc10f060 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009100 tx=0x7fc4d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:55.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.288+0000 7fc4d8ff9700 1 -- 172.21.15.38:0/1889232386 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4d000e040 con 0x7fc4dc102730 2024-04-22T21:06:55.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.288+0000 7fc4d8ff9700 1 -- 172.21.15.38:0/1889232386 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc4d0013420 con 0x7fc4dc102730 2024-04-22T21:06:55.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.288+0000 7fc4e23ce700 1 -- 172.21.15.38:0/1889232386 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4dc10f820 con 0x7fc4dc102730 2024-04-22T21:06:55.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.288+0000 7fc4d8ff9700 1 -- 172.21.15.38:0/1889232386 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4d001c610 con 0x7fc4dc102730 2024-04-22T21:06:55.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.288+0000 7fc4e23ce700 1 -- 172.21.15.38:0/1889232386 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4dc117e80 con 0x7fc4dc102730 2024-04-22T21:06:55.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.289+0000 7fc4e23ce700 1 -- 172.21.15.38:0/1889232386 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4dc104260 con 0x7fc4dc102730 2024-04-22T21:06:55.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.289+0000 7fc4d8ff9700 1 -- 172.21.15.38:0/1889232386 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fc4d001c840 con 0x7fc4dc102730 2024-04-22T21:06:55.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.290+0000 7fc4d8ff9700 1 --2- 172.21.15.38:0/1889232386 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc4c40692a0 0x7fc4c406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:55.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.290+0000 7fc4dbfff700 1 --2- 172.21.15.38:0/1889232386 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc4c40692a0 0x7fc4c406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:55.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.290+0000 7fc4d8ff9700 1 -- 172.21.15.38:0/1889232386 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc4d0088020 con 0x7fc4dc102730 2024-04-22T21:06:55.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.290+0000 7fc4dbfff700 1 --2- 172.21.15.38:0/1889232386 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc4c40692a0 0x7fc4c406b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc0041d0 tx=0x7fc4cc009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:55.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.294+0000 7fc4d8ff9700 1 -- 172.21.15.38:0/1889232386 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fc4d0054730 con 0x7fc4dc102730 2024-04-22T21:06:55.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.504+0000 7fc4e23ce700 1 -- 172.21.15.38:0/1889232386 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc4dc108e70 con 0x7fc4dc102730 2024-04-22T21:06:55.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.505+0000 7fc4d8ff9700 1 -- 172.21.15.38:0/1889232386 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc4d0057d50 con 0x7fc4dc102730 2024-04-22T21:06:55.501 INFO:teuthology.orchestra.run.smithi038.stdout:81604378651 2024-04-22T21:06:55.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.507+0000 7fc4e23ce700 1 -- 172.21.15.38:0/1889232386 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc4c40692a0 msgr2=0x7fc4c406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:55.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.507+0000 7fc4e23ce700 1 --2- 172.21.15.38:0/1889232386 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc4c40692a0 0x7fc4c406b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc0041d0 tx=0x7fc4cc009040 comp rx=0 tx=0).stop 2024-04-22T21:06:55.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.507+0000 7fc4e23ce700 1 -- 172.21.15.38:0/1889232386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4dc102730 msgr2=0x7fc4dc10f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:55.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.507+0000 7fc4e23ce700 1 --2- 172.21.15.38:0/1889232386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4dc102730 0x7fc4dc10f060 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009100 tx=0x7fc4d0009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:55.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.507+0000 7fc4e23ce700 1 -- 172.21.15.38:0/1889232386 shutdown_connections 2024-04-22T21:06:55.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.507+0000 7fc4e23ce700 1 --2- 172.21.15.38:0/1889232386 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc4c40692a0 0x7fc4c406b750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:55.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.507+0000 7fc4e23ce700 1 --2- 172.21.15.38:0/1889232386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4dc101d90 0x7fc4dc10eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:55.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.507+0000 7fc4e23ce700 1 --2- 172.21.15.38:0/1889232386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4dc102730 0x7fc4dc10f060 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:55.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.507+0000 7fc4e23ce700 1 -- 172.21.15.38:0/1889232386 >> 172.21.15.38:0/1889232386 conn(0x7fc4dc0f7900 msgr2=0x7fc4dc101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:55.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.507+0000 7fc4e23ce700 1 -- 172.21.15.38:0/1889232386 shutdown_connections 2024-04-22T21:06:55.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:55.507+0000 7fc4e23ce700 1 -- 172.21.15.38:0/1889232386 wait complete. 2024-04-22T21:06:55.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:55 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1889232386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T21:06:55.999 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:55 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/1889232386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T21:06:56.306 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378651 for osd.2 2024-04-22T21:06:56.307 DEBUG:teuthology.parallel:result is None 2024-04-22T21:06:56.865 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:56 smithi038 ceph-mon[26374]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:56.999 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:56 smithi098 ceph-mon[34053]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:57.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.132+0000 7f7bcb2e6700 1 -- 172.21.15.38:0/3442260625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc4101d20 msgr2=0x7f7bc41020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.132+0000 7f7bcb2e6700 1 --2- 172.21.15.38:0/3442260625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc4101d20 0x7f7bc41020f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0004940 tx=0x7f7bc0009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.132+0000 7f7bcb2e6700 1 -- 172.21.15.38:0/3442260625 shutdown_connections 2024-04-22T21:06:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.132+0000 7f7bcb2e6700 1 --2- 172.21.15.38:0/3442260625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc41026c0 0x7f7bc40fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.132+0000 7f7bcb2e6700 1 --2- 172.21.15.38:0/3442260625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc4101d20 0x7f7bc41020f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.132+0000 7f7bcb2e6700 1 -- 172.21.15.38:0/3442260625 >> 172.21.15.38:0/3442260625 conn(0x7f7bc40f7870 msgr2=0x7f7bc40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.132+0000 7f7bcb2e6700 1 -- 172.21.15.38:0/3442260625 shutdown_connections 2024-04-22T21:06:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.132+0000 7f7bcb2e6700 1 -- 172.21.15.38:0/3442260625 wait complete. 2024-04-22T21:06:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.132+0000 7f7bcb2e6700 1 Processor -- start 2024-04-22T21:06:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.132+0000 7f7bcb2e6700 1 -- start start 2024-04-22T21:06:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bcb2e6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc4101d20 0x7f7bc410eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bcb2e6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc41026c0 0x7f7bc410f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bcb2e6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc410f5f0 con 0x7f7bc4101d20 2024-04-22T21:06:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bcb2e6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc410f730 con 0x7f7bc41026c0 2024-04-22T21:06:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bc9082700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc4101d20 0x7f7bc410eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bc9082700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc4101d20 0x7f7bc410eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51484/0 (socket says 172.21.15.38:51484) 2024-04-22T21:06:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bc9082700 1 -- 172.21.15.38:0/2407738933 learned_addr learned my addr 172.21.15.38:0/2407738933 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bc8881700 1 --2- 172.21.15.38:0/2407738933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc41026c0 0x7f7bc410f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bc9082700 1 -- 172.21.15.38:0/2407738933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc41026c0 msgr2=0x7f7bc410f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bc9082700 1 --2- 172.21.15.38:0/2407738933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc41026c0 0x7f7bc410f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bc9082700 1 -- 172.21.15.38:0/2407738933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc0005040 con 0x7f7bc4101d20 2024-04-22T21:06:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bc8881700 1 --2- 172.21.15.38:0/2407738933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc41026c0 0x7f7bc410f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T21:06:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bc9082700 1 --2- 172.21.15.38:0/2407738933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc4101d20 0x7f7bc410eb70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7bc000b3b0 tx=0x7f7bc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bb9ffb700 1 -- 172.21.15.38:0/2407738933 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bc000e040 con 0x7f7bc4101d20 2024-04-22T21:06:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bb9ffb700 1 -- 172.21.15.38:0/2407738933 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7bc000bbc0 con 0x7f7bc4101d20 2024-04-22T21:06:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bcb2e6700 1 -- 172.21.15.38:0/2407738933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc410f9b0 con 0x7f7bc4101d20 2024-04-22T21:06:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.133+0000 7f7bb9ffb700 1 -- 172.21.15.38:0/2407738933 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bc001c3f0 con 0x7f7bc4101d20 2024-04-22T21:06:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.134+0000 7f7bcb2e6700 1 -- 172.21.15.38:0/2407738933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc4117ee0 con 0x7f7bc4101d20 2024-04-22T21:06:57.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.134+0000 7f7bcb2e6700 1 -- 172.21.15.38:0/2407738933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc40fd470 con 0x7f7bc4101d20 2024-04-22T21:06:57.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.135+0000 7f7bb9ffb700 1 -- 172.21.15.38:0/2407738933 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f7bc000bd30 con 0x7f7bc4101d20 2024-04-22T21:06:57.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.135+0000 7f7bb9ffb700 1 --2- 172.21.15.38:0/2407738933 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7bb00692a0 0x7f7bb006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:57.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.136+0000 7f7bb9ffb700 1 -- 172.21.15.38:0/2407738933 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7bc0088030 con 0x7f7bc4101d20 2024-04-22T21:06:57.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.136+0000 7f7bc8881700 1 --2- 172.21.15.38:0/2407738933 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7bb00692a0 0x7f7bb006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:57.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.136+0000 7f7bc8881700 1 --2- 172.21.15.38:0/2407738933 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7bb00692a0 0x7f7bb006b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009fd0 tx=0x7f7bb4009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:57.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.141+0000 7f7bb9ffb700 1 -- 172.21.15.38:0/2407738933 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f7bc0054750 con 0x7f7bc4101d20 2024-04-22T21:06:57.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.337+0000 7f7bcb2e6700 1 -- 172.21.15.38:0/2407738933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f7bc4118150 con 0x7f7bc4101d20 2024-04-22T21:06:57.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.337+0000 7f7bb9ffb700 1 -- 172.21.15.38:0/2407738933 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7bc0057d70 con 0x7f7bc4101d20 2024-04-22T21:06:57.334 INFO:teuthology.orchestra.run.smithi038.stdout:171798691852 2024-04-22T21:06:57.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.339+0000 7f7bcb2e6700 1 -- 172.21.15.38:0/2407738933 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7bb00692a0 msgr2=0x7f7bb006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:57.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.339+0000 7f7bcb2e6700 1 --2- 172.21.15.38:0/2407738933 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7bb00692a0 0x7f7bb006b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009fd0 tx=0x7f7bb4009ee0 comp rx=0 tx=0).stop 2024-04-22T21:06:57.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.339+0000 7f7bcb2e6700 1 -- 172.21.15.38:0/2407738933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc4101d20 msgr2=0x7f7bc410eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:57.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.339+0000 7f7bcb2e6700 1 --2- 172.21.15.38:0/2407738933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc4101d20 0x7f7bc410eb70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7bc000b3b0 tx=0x7f7bc0009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:57.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.339+0000 7f7bcb2e6700 1 -- 172.21.15.38:0/2407738933 shutdown_connections 2024-04-22T21:06:57.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.339+0000 7f7bcb2e6700 1 --2- 172.21.15.38:0/2407738933 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f7bb00692a0 0x7f7bb006b750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:57.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.339+0000 7f7bcb2e6700 1 --2- 172.21.15.38:0/2407738933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc4101d20 0x7f7bc410eb70 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:57.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.339+0000 7f7bcb2e6700 1 --2- 172.21.15.38:0/2407738933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc41026c0 0x7f7bc410f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:57.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.339+0000 7f7bcb2e6700 1 -- 172.21.15.38:0/2407738933 >> 172.21.15.38:0/2407738933 conn(0x7f7bc40f7870 msgr2=0x7f7bc40f8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:57.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.339+0000 7f7bcb2e6700 1 -- 172.21.15.38:0/2407738933 shutdown_connections 2024-04-22T21:06:57.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:57.339+0000 7f7bcb2e6700 1 -- 172.21.15.38:0/2407738933 wait complete. 2024-04-22T21:06:57.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:57 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/2407738933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T21:06:57.973 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691852 got 171798691852 for osd.5 2024-04-22T21:06:57.974 DEBUG:teuthology.parallel:result is None 2024-04-22T21:06:57.974 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-22T21:06:57.974 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T21:06:57.999 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:57 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/2407738933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T21:06:58.242 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:06:58.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:06:58 smithi038 ceph-mon[26374]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:58.999 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:06:58 smithi098 ceph-mon[34053]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:06:59.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.741+0000 7fae1189f700 1 -- 172.21.15.38:0/1719973649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0c1037c0 msgr2=0x7fae0c0fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:59.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.741+0000 7fae1189f700 1 --2- 172.21.15.38:0/1719973649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0c1037c0 0x7fae0c0fce30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fadf4004940 tx=0x7fadf4009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:59.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.741+0000 7fae1189f700 1 -- 172.21.15.38:0/1719973649 shutdown_connections 2024-04-22T21:06:59.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.741+0000 7fae1189f700 1 --2- 172.21.15.38:0/1719973649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0c1037c0 0x7fae0c0fce30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:59.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.741+0000 7fae1189f700 1 --2- 172.21.15.38:0/1719973649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae0c102e20 0x7fae0c1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:59.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.741+0000 7fae1189f700 1 -- 172.21.15.38:0/1719973649 >> 172.21.15.38:0/1719973649 conn(0x7fae0c0f89b0 msgr2=0x7fae0c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:59.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.741+0000 7fae1189f700 1 -- 172.21.15.38:0/1719973649 shutdown_connections 2024-04-22T21:06:59.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.741+0000 7fae1189f700 1 -- 172.21.15.38:0/1719973649 wait complete. 2024-04-22T21:06:59.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.742+0000 7fae1189f700 1 Processor -- start 2024-04-22T21:06:59.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.742+0000 7fae1189f700 1 -- start start 2024-04-22T21:06:59.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.742+0000 7fae1189f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae0c102e20 0x7fae0c1139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:59.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.742+0000 7fae1189f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0c1037c0 0x7fae0c113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:59.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.742+0000 7fae1189f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c114530 con 0x7fae0c1037c0 2024-04-22T21:06:59.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.742+0000 7fae1189f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c10db50 con 0x7fae0c102e20 2024-04-22T21:06:59.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.742+0000 7fae0a7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0c1037c0 0x7fae0c113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:59.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.742+0000 7fae0a7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0c1037c0 0x7fae0c113ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51508/0 (socket says 172.21.15.38:51508) 2024-04-22T21:06:59.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.742+0000 7fae0a7fc700 1 -- 172.21.15.38:0/3524315905 learned_addr learned my addr 172.21.15.38:0/3524315905 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:06:59.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.742+0000 7fae0affd700 1 --2- 172.21.15.38:0/3524315905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae0c102e20 0x7fae0c1139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:59.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.742+0000 7fae0a7fc700 1 -- 172.21.15.38:0/3524315905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae0c102e20 msgr2=0x7fae0c1139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:59.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.742+0000 7fae0a7fc700 1 --2- 172.21.15.38:0/3524315905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae0c102e20 0x7fae0c1139a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:59.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.742+0000 7fae0a7fc700 1 -- 172.21.15.38:0/3524315905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf4005040 con 0x7fae0c1037c0 2024-04-22T21:06:59.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.743+0000 7fae0a7fc700 1 --2- 172.21.15.38:0/3524315905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0c1037c0 0x7fae0c113ee0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fadf4009180 tx=0x7fadf4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:59.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.743+0000 7fae037fe700 1 -- 172.21.15.38:0/3524315905 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadf400e040 con 0x7fae0c1037c0 2024-04-22T21:06:59.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.743+0000 7fae1189f700 1 -- 172.21.15.38:0/3524315905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0c10ddd0 con 0x7fae0c1037c0 2024-04-22T21:06:59.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.743+0000 7fae037fe700 1 -- 172.21.15.38:0/3524315905 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fadf4013420 con 0x7fae0c1037c0 2024-04-22T21:06:59.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.743+0000 7fae037fe700 1 -- 172.21.15.38:0/3524315905 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadf401c610 con 0x7fae0c1037c0 2024-04-22T21:06:59.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.743+0000 7fae1189f700 1 -- 172.21.15.38:0/3524315905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0c10e390 con 0x7fae0c1037c0 2024-04-22T21:06:59.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.744+0000 7fae1189f700 1 -- 172.21.15.38:0/3524315905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae0c1052f0 con 0x7fae0c1037c0 2024-04-22T21:06:59.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.744+0000 7fae037fe700 1 -- 172.21.15.38:0/3524315905 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fadf4013a00 con 0x7fae0c1037c0 2024-04-22T21:06:59.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.745+0000 7fae037fe700 1 --2- 172.21.15.38:0/3524315905 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fadf80692a0 0x7fadf806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:06:59.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.745+0000 7fae037fe700 1 -- 172.21.15.38:0/3524315905 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fadf4087f80 con 0x7fae0c1037c0 2024-04-22T21:06:59.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.745+0000 7fae0affd700 1 --2- 172.21.15.38:0/3524315905 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fadf80692a0 0x7fadf806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:06:59.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.745+0000 7fae0affd700 1 --2- 172.21.15.38:0/3524315905 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fadf80692a0 0x7fadf806b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fae04004fd0 tx=0x7fae04009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:06:59.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.749+0000 7fae037fe700 1 -- 172.21.15.38:0/3524315905 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fadf40545f0 con 0x7fae0c1037c0 2024-04-22T21:06:59.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.938+0000 7fae1189f700 1 -- 172.21.15.38:0/3524315905 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fae0c10eff0 con 0x7fadf80692a0 2024-04-22T21:06:59.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.939+0000 7fae037fe700 1 -- 172.21.15.38:0/3524315905 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28225 (secure 0 0 0) 0x7fae0c10eff0 con 0x7fadf80692a0 2024-04-22T21:06:59.935 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:06:59.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.941+0000 7fae1189f700 1 -- 172.21.15.38:0/3524315905 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fadf80692a0 msgr2=0x7fadf806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:59.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.941+0000 7fae1189f700 1 --2- 172.21.15.38:0/3524315905 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fadf80692a0 0x7fadf806b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fae04004fd0 tx=0x7fae04009040 comp rx=0 tx=0).stop 2024-04-22T21:06:59.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.941+0000 7fae1189f700 1 -- 172.21.15.38:0/3524315905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0c1037c0 msgr2=0x7fae0c113ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:06:59.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.941+0000 7fae1189f700 1 --2- 172.21.15.38:0/3524315905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0c1037c0 0x7fae0c113ee0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fadf4009180 tx=0x7fadf4009e30 comp rx=0 tx=0).stop 2024-04-22T21:06:59.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.941+0000 7fae1189f700 1 -- 172.21.15.38:0/3524315905 shutdown_connections 2024-04-22T21:06:59.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.941+0000 7fae1189f700 1 --2- 172.21.15.38:0/3524315905 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fadf80692a0 0x7fadf806b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:59.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.941+0000 7fae1189f700 1 --2- 172.21.15.38:0/3524315905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae0c102e20 0x7fae0c1139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:59.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.941+0000 7fae1189f700 1 --2- 172.21.15.38:0/3524315905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0c1037c0 0x7fae0c113ee0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:06:59.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.941+0000 7fae1189f700 1 -- 172.21.15.38:0/3524315905 >> 172.21.15.38:0/3524315905 conn(0x7fae0c0f89b0 msgr2=0x7fae0c1021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:06:59.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.941+0000 7fae1189f700 1 -- 172.21.15.38:0/3524315905 shutdown_connections 2024-04-22T21:06:59.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:06:59.941+0000 7fae1189f700 1 -- 172.21.15.38:0/3524315905 wait complete. 2024-04-22T21:06:59.938 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-04-22T21:07:00.721 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-04-22T21:06:58.085647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788084,"kb_used_data":3924,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682828,"statfs":{"total":575970213888,"available":574139215872,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002191"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":188,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T21:06:10.236722+0000","last_change":"2024-04-22T21:05:45.003181+0000","last_active":"2024-04-22T21:06:10.236722+0000","last_peered":"2024-04-22T21:06:10.236722+0000","last_clean":"2024-04-22T21:06:10.236722+0000","last_became_active":"2024-04-22T21:05:45.002845+0000","last_became_peered":"2024-04-22T21:05:45.002845+0000","last_unstale":"2024-04-22T21:06:10.236722+0000","last_undegraded":"2024-04-22T21:06:10.236722+0000","last_fullsized":"2024-04-22T21:06:10.236722+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T21:04:56.607700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T21:04:56.607700+0000","last_clean_scrub_stamp":"2024-04-22T21:04:56.607700+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T05:29:08.855755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]}]},{"osd":4,"up_from":35,"seq":150323855377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297724,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447428,"statfs":{"total":95995035648,"available":95690166272,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.44800000000000001}]},{"osd":1,"last update":"Mon Apr 22 21:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.46500000000000002}]},{"osd":2,"last update":"Mon Apr 22 21:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.502},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.49399999999999999}]},{"osd":3,"last update":"Mon Apr 22 21:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.377},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.48099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117014,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.59099999999999997}]},{"osd":1,"last update":"Mon Apr 22 21:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.499}]},{"osd":2,"last update":"Mon Apr 22 21:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.67600000000000005}]},{"osd":4,"last update":"Mon Apr 22 21:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297724,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447428,"statfs":{"total":95995035648,"available":95690166272,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.249},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.26400000000000001}]},{"osd":1,"last update":"Mon Apr 22 21:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.29599999999999999}]},{"osd":3,"last update":"Mon Apr 22 21:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.51700000000000002}]},{"osd":4,"last update":"Mon Apr 22 21:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.56499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298308,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446844,"statfs":{"total":95995035648,"available":95689568256,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 21:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.32400000000000001,"15min":0.31900000000000001},"min":{"1min":0.26300000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.497,"5min":0.497,"15min":0.497},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.34699999999999998,"15min":0.34100000000000003},"min":{"1min":0.249,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.36799999999999999}]},{"osd":2,"last update":"Mon Apr 22 21:05:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.38600000000000001}]},{"osd":3,"last update":"Mon Apr 22 21:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.60099999999999998}]},{"osd":4,"last update":"Mon Apr 22 21:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.626},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.57699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298308,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446844,"statfs":{"total":95995035648,"available":95689568256,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.33500000000000002,"15min":0.33000000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.377},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.36499999999999999,"15min":0.36099999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.31}]},{"osd":2,"last update":"Mon Apr 22 21:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.36299999999999999}]},{"osd":3,"last update":"Mon Apr 22 21:06:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.53100000000000003}]},{"osd":4,"last update":"Mon Apr 22 21:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.50900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T21:07:00.722 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T21:07:00.994 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:01.373 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:01 smithi038 ceph-mon[26374]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:07:01.373 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:01 smithi038 ceph-mon[26374]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:01.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:01 smithi098 ceph-mon[34053]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:07:01.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:01 smithi098 ceph-mon[34053]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:02.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.319+0000 7f54fada7700 1 -- 172.21.15.38:0/2031765753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f4101d20 msgr2=0x7f54f41020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:02.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.319+0000 7f54fada7700 1 --2- 172.21.15.38:0/2031765753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f4101d20 0x7f54f41020f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f54e8004940 tx=0x7f54e8009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:02.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.319+0000 7f54fada7700 1 -- 172.21.15.38:0/2031765753 shutdown_connections 2024-04-22T21:07:02.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.319+0000 7f54fada7700 1 --2- 172.21.15.38:0/2031765753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54f40fbd20 0x7f54f40fc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:02.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.319+0000 7f54fada7700 1 --2- 172.21.15.38:0/2031765753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f4101d20 0x7f54f41020f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:02.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.319+0000 7f54fada7700 1 -- 172.21.15.38:0/2031765753 >> 172.21.15.38:0/2031765753 conn(0x7f54f4071ca0 msgr2=0x7f54f40720a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:02.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.319+0000 7f54fada7700 1 -- 172.21.15.38:0/2031765753 shutdown_connections 2024-04-22T21:07:02.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.319+0000 7f54fada7700 1 -- 172.21.15.38:0/2031765753 wait complete. 2024-04-22T21:07:02.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.320+0000 7f54fada7700 1 Processor -- start 2024-04-22T21:07:02.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.320+0000 7f54fada7700 1 -- start start 2024-04-22T21:07:02.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.320+0000 7f54fada7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54f40fbd20 0x7f54f4113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:02.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.320+0000 7f54fada7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f4101d20 0x7f54f4114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:02.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.320+0000 7f54fada7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f4114700 con 0x7f54f4101d20 2024-04-22T21:07:02.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.320+0000 7f54fada7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f410da60 con 0x7f54f40fbd20 2024-04-22T21:07:02.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.320+0000 7f54f3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f4101d20 0x7f54f4114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:02.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.320+0000 7f54f3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f4101d20 0x7f54f4114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51520/0 (socket says 172.21.15.38:51520) 2024-04-22T21:07:02.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.320+0000 7f54f3fff700 1 -- 172.21.15.38:0/1952467774 learned_addr learned my addr 172.21.15.38:0/1952467774 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:02.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.321+0000 7f54f8b43700 1 --2- 172.21.15.38:0/1952467774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54f40fbd20 0x7f54f4113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:02.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.321+0000 7f54f3fff700 1 -- 172.21.15.38:0/1952467774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54f40fbd20 msgr2=0x7f54f4113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:02.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.321+0000 7f54f3fff700 1 --2- 172.21.15.38:0/1952467774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54f40fbd20 0x7f54f4113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:02.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.321+0000 7f54f3fff700 1 -- 172.21.15.38:0/1952467774 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e8005040 con 0x7f54f4101d20 2024-04-22T21:07:02.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.321+0000 7f54f3fff700 1 --2- 172.21.15.38:0/1952467774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f4101d20 0x7f54f4114020 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f54e000b790 tx=0x7f54e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:02.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.321+0000 7f54f17fa700 1 -- 172.21.15.38:0/1952467774 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54e000f6f0 con 0x7f54f4101d20 2024-04-22T21:07:02.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.321+0000 7f54f17fa700 1 -- 172.21.15.38:0/1952467774 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54e000fd20 con 0x7f54f4101d20 2024-04-22T21:07:02.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.321+0000 7f54fada7700 1 -- 172.21.15.38:0/1952467774 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f410dd40 con 0x7f54f4101d20 2024-04-22T21:07:02.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.321+0000 7f54f17fa700 1 -- 172.21.15.38:0/1952467774 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54e00103a0 con 0x7f54f4101d20 2024-04-22T21:07:02.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.321+0000 7f54fada7700 1 -- 172.21.15.38:0/1952467774 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f410e300 con 0x7f54f4101d20 2024-04-22T21:07:02.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.322+0000 7f54fada7700 1 -- 172.21.15.38:0/1952467774 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f40fd3c0 con 0x7f54f4101d20 2024-04-22T21:07:02.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.322+0000 7f54f17fa700 1 -- 172.21.15.38:0/1952467774 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f54e0011400 con 0x7f54f4101d20 2024-04-22T21:07:02.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.323+0000 7f54f17fa700 1 --2- 172.21.15.38:0/1952467774 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f54e40692b0 0x7f54e406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:02.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.323+0000 7f54f17fa700 1 -- 172.21.15.38:0/1952467774 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f54e0086b60 con 0x7f54f4101d20 2024-04-22T21:07:02.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.323+0000 7f54f8b43700 1 --2- 172.21.15.38:0/1952467774 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f54e40692b0 0x7f54e406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:02.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.323+0000 7f54f8b43700 1 --2- 172.21.15.38:0/1952467774 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f54e40692b0 0x7f54e406b760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f54e800b3b0 tx=0x7f54e8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:02.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.327+0000 7f54f17fa700 1 -- 172.21.15.38:0/1952467774 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f54e0056d00 con 0x7f54f4101d20 2024-04-22T21:07:02.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.517+0000 7f54fada7700 1 -- 172.21.15.38:0/1952467774 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f54f4108e00 con 0x7f54e40692b0 2024-04-22T21:07:02.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.518+0000 7f54f17fa700 1 -- 172.21.15.38:0/1952467774 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28210 (secure 0 0 0) 0x7f54f4108e00 con 0x7f54e40692b0 2024-04-22T21:07:02.514 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:07:02.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.520+0000 7f54fada7700 1 -- 172.21.15.38:0/1952467774 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f54e40692b0 msgr2=0x7f54e406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:02.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.520+0000 7f54fada7700 1 --2- 172.21.15.38:0/1952467774 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f54e40692b0 0x7f54e406b760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f54e800b3b0 tx=0x7f54e8009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:02.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.520+0000 7f54fada7700 1 -- 172.21.15.38:0/1952467774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f4101d20 msgr2=0x7f54f4114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:02.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.520+0000 7f54fada7700 1 --2- 172.21.15.38:0/1952467774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f4101d20 0x7f54f4114020 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f54e000b790 tx=0x7f54e0009250 comp rx=0 tx=0).stop 2024-04-22T21:07:02.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.520+0000 7f54fada7700 1 -- 172.21.15.38:0/1952467774 shutdown_connections 2024-04-22T21:07:02.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.520+0000 7f54fada7700 1 --2- 172.21.15.38:0/1952467774 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f54e40692b0 0x7f54e406b760 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:02.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.520+0000 7f54fada7700 1 --2- 172.21.15.38:0/1952467774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54f40fbd20 0x7f54f4113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:02.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.520+0000 7f54fada7700 1 --2- 172.21.15.38:0/1952467774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f4101d20 0x7f54f4114020 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:02.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.520+0000 7f54fada7700 1 -- 172.21.15.38:0/1952467774 >> 172.21.15.38:0/1952467774 conn(0x7f54f4071ca0 msgr2=0x7f54f40fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:02.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.520+0000 7f54fada7700 1 -- 172.21.15.38:0/1952467774 shutdown_connections 2024-04-22T21:07:02.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:02.520+0000 7f54fada7700 1 -- 172.21.15.38:0/1952467774 wait complete. 2024-04-22T21:07:02.518 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-04-22T21:07:03.298 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-04-22T21:07:02.086237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788084,"kb_used_data":3924,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682828,"statfs":{"total":575970213888,"available":574139215872,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001957"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":188,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T21:06:10.236722+0000","last_change":"2024-04-22T21:05:45.003181+0000","last_active":"2024-04-22T21:06:10.236722+0000","last_peered":"2024-04-22T21:06:10.236722+0000","last_clean":"2024-04-22T21:06:10.236722+0000","last_became_active":"2024-04-22T21:05:45.002845+0000","last_became_peered":"2024-04-22T21:05:45.002845+0000","last_unstale":"2024-04-22T21:06:10.236722+0000","last_undegraded":"2024-04-22T21:06:10.236722+0000","last_fullsized":"2024-04-22T21:06:10.236722+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T21:04:56.607700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T21:04:56.607700+0000","last_clean_scrub_stamp":"2024-04-22T21:04:56.607700+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T05:29:08.855755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297724,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447428,"statfs":{"total":95995035648,"available":95690166272,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.44800000000000001}]},{"osd":1,"last update":"Mon Apr 22 21:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.46500000000000002}]},{"osd":2,"last update":"Mon Apr 22 21:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.502},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.49399999999999999}]},{"osd":3,"last update":"Mon Apr 22 21:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.377},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.48099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.42599999999999999}]},{"osd":1,"last update":"Mon Apr 22 21:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.35999999999999999}]},{"osd":2,"last update":"Mon Apr 22 21:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.377},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.505}]},{"osd":4,"last update":"Mon Apr 22 21:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.34200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297724,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447428,"statfs":{"total":95995035648,"available":95690166272,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.36099999999999999,"15min":0.36699999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.377,"15min":0.38500000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.30399999999999999}]},{"osd":1,"last update":"Mon Apr 22 21:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.40100000000000002,"15min":0.40899999999999997},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.39800000000000002,"15min":0.40699999999999997},"min":{"1min":0.27100000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53600000000000003,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.35099999999999998}]},{"osd":3,"last update":"Mon Apr 22 21:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.63900000000000001}]},{"osd":4,"last update":"Mon Apr 22 21:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":6.3650000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":6.3769999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298308,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446844,"statfs":{"total":95995035648,"available":95689568256,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 21:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.32400000000000001,"15min":0.31900000000000001},"min":{"1min":0.26300000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.497,"5min":0.497,"15min":0.497},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.34699999999999998,"15min":0.34100000000000003},"min":{"1min":0.249,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.35199999999999998}]},{"osd":2,"last update":"Mon Apr 22 21:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.39100000000000001,"15min":0.40200000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53500000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.40000000000000002,"15min":0.40699999999999997},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.498,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.34000000000000002}]},{"osd":3,"last update":"Mon Apr 22 21:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.623},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.53200000000000003}]},{"osd":4,"last update":"Mon Apr 22 21:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.66200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298308,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446844,"statfs":{"total":95995035648,"available":95689568256,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.33500000000000002,"15min":0.33000000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.377},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.36499999999999999,"15min":0.36099999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.31}]},{"osd":2,"last update":"Mon Apr 22 21:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.36299999999999999}]},{"osd":3,"last update":"Mon Apr 22 21:06:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.53100000000000003}]},{"osd":4,"last update":"Mon Apr 22 21:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.50900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T21:07:03.299 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-22T21:07:03.299 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T21:07:03.300 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-22T21:07:03.300 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-22T21:07:03.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:03 smithi098 ceph-mon[34053]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:03.570 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:03.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:03 smithi038 ceph-mon[26374]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:04.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:04 smithi038 ceph-mon[26374]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:07:04.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:04 smithi098 ceph-mon[34053]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:07:04.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.879+0000 7fd6dc30b700 1 -- 172.21.15.38:0/3452452945 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4102e10 msgr2=0x7fd6d41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:04.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.879+0000 7fd6dc30b700 1 --2- 172.21.15.38:0/3452452945 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4102e10 0x7fd6d41031e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009ce0 tx=0x7fd6d0004d10 comp rx=0 tx=0).stop 2024-04-22T21:07:04.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.879+0000 7fd6dc30b700 1 -- 172.21.15.38:0/3452452945 shutdown_connections 2024-04-22T21:07:04.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.879+0000 7fd6dc30b700 1 --2- 172.21.15.38:0/3452452945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6d41037b0 0x7fd6d40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:04.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.879+0000 7fd6dc30b700 1 --2- 172.21.15.38:0/3452452945 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4102e10 0x7fd6d41031e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:04.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.879+0000 7fd6dc30b700 1 -- 172.21.15.38:0/3452452945 >> 172.21.15.38:0/3452452945 conn(0x7fd6d40f89a0 msgr2=0x7fd6d40fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:04.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.879+0000 7fd6dc30b700 1 -- 172.21.15.38:0/3452452945 shutdown_connections 2024-04-22T21:07:04.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.879+0000 7fd6dc30b700 1 -- 172.21.15.38:0/3452452945 wait complete. 2024-04-22T21:07:04.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.880+0000 7fd6dc30b700 1 Processor -- start 2024-04-22T21:07:04.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.880+0000 7fd6dc30b700 1 -- start start 2024-04-22T21:07:04.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.880+0000 7fd6dc30b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6d4102e10 0x7fd6d4113d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:04.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.880+0000 7fd6dc30b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d41037b0 0x7fd6d4114240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:04.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.880+0000 7fd6dc30b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d4114890 con 0x7fd6d41037b0 2024-04-22T21:07:04.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.880+0000 7fd6dc30b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d410dca0 con 0x7fd6d4102e10 2024-04-22T21:07:04.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.880+0000 7fd6d98a6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d41037b0 0x7fd6d4114240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:04.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.880+0000 7fd6d98a6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d41037b0 0x7fd6d4114240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50328/0 (socket says 172.21.15.38:50328) 2024-04-22T21:07:04.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.880+0000 7fd6d98a6700 1 -- 172.21.15.38:0/1198849480 learned_addr learned my addr 172.21.15.38:0/1198849480 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:04.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.880+0000 7fd6da0a7700 1 --2- 172.21.15.38:0/1198849480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6d4102e10 0x7fd6d4113d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:04.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.880+0000 7fd6d98a6700 1 -- 172.21.15.38:0/1198849480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6d4102e10 msgr2=0x7fd6d4113d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:04.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.880+0000 7fd6d98a6700 1 --2- 172.21.15.38:0/1198849480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6d4102e10 0x7fd6d4113d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:04.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.880+0000 7fd6d98a6700 1 -- 172.21.15.38:0/1198849480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d0009940 con 0x7fd6d41037b0 2024-04-22T21:07:04.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.880+0000 7fd6da0a7700 1 --2- 172.21.15.38:0/1198849480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6d4102e10 0x7fd6d4113d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T21:07:04.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.881+0000 7fd6d98a6700 1 --2- 172.21.15.38:0/1198849480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d41037b0 0x7fd6d4114240 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd6c400c800 tx=0x7fd6c4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:04.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.881+0000 7fd6caffd700 1 -- 172.21.15.38:0/1198849480 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6c400a760 con 0x7fd6d41037b0 2024-04-22T21:07:04.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.881+0000 7fd6caffd700 1 -- 172.21.15.38:0/1198849480 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6c400ad90 con 0x7fd6d41037b0 2024-04-22T21:07:04.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.881+0000 7fd6dc30b700 1 -- 172.21.15.38:0/1198849480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d410df80 con 0x7fd6d41037b0 2024-04-22T21:07:04.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.881+0000 7fd6caffd700 1 -- 172.21.15.38:0/1198849480 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6c40113a0 con 0x7fd6d41037b0 2024-04-22T21:07:04.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.881+0000 7fd6dc30b700 1 -- 172.21.15.38:0/1198849480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d410e4c0 con 0x7fd6d41037b0 2024-04-22T21:07:04.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.882+0000 7fd6dc30b700 1 -- 172.21.15.38:0/1198849480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d4100120 con 0x7fd6d41037b0 2024-04-22T21:07:04.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.882+0000 7fd6caffd700 1 -- 172.21.15.38:0/1198849480 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fd6c4012400 con 0x7fd6d41037b0 2024-04-22T21:07:04.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.883+0000 7fd6caffd700 1 --2- 172.21.15.38:0/1198849480 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fd6c00692a0 0x7fd6c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:04.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.883+0000 7fd6caffd700 1 -- 172.21.15.38:0/1198849480 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd6c4086ea0 con 0x7fd6d41037b0 2024-04-22T21:07:04.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.883+0000 7fd6da0a7700 1 --2- 172.21.15.38:0/1198849480 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fd6c00692a0 0x7fd6c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:04.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.883+0000 7fd6da0a7700 1 --2- 172.21.15.38:0/1198849480 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fd6c00692a0 0x7fd6c006b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00048c0 tx=0x7fd6d0004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:04.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:04.887+0000 7fd6caffd700 1 -- 172.21.15.38:0/1198849480 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd6c40534f0 con 0x7fd6d41037b0 2024-04-22T21:07:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:05.124+0000 7fd6dc30b700 1 -- 172.21.15.38:0/1198849480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd6d410f390 con 0x7fd6d41037b0 2024-04-22T21:07:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:05.124+0000 7fd6caffd700 1 -- 172.21.15.38:0/1198849480 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fd6c4056b10 con 0x7fd6d41037b0 2024-04-22T21:07:05.120 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:07:05.120 INFO:teuthology.orchestra.run.smithi038.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T21:07:05.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:05.126+0000 7fd6dc30b700 1 -- 172.21.15.38:0/1198849480 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fd6c00692a0 msgr2=0x7fd6c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:05.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:05.126+0000 7fd6dc30b700 1 --2- 172.21.15.38:0/1198849480 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fd6c00692a0 0x7fd6c006b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00048c0 tx=0x7fd6d0004d10 comp rx=0 tx=0).stop 2024-04-22T21:07:05.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:05.126+0000 7fd6dc30b700 1 -- 172.21.15.38:0/1198849480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d41037b0 msgr2=0x7fd6d4114240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:05.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:05.126+0000 7fd6dc30b700 1 --2- 172.21.15.38:0/1198849480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d41037b0 0x7fd6d4114240 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd6c400c800 tx=0x7fd6c4004470 comp rx=0 tx=0).stop 2024-04-22T21:07:05.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:05.126+0000 7fd6dc30b700 1 -- 172.21.15.38:0/1198849480 shutdown_connections 2024-04-22T21:07:05.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:05.126+0000 7fd6dc30b700 1 --2- 172.21.15.38:0/1198849480 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fd6c00692a0 0x7fd6c006b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:05.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:05.126+0000 7fd6dc30b700 1 --2- 172.21.15.38:0/1198849480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6d4102e10 0x7fd6d4113d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:05.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:05.126+0000 7fd6dc30b700 1 --2- 172.21.15.38:0/1198849480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d41037b0 0x7fd6d4114240 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:05.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:05.126+0000 7fd6dc30b700 1 -- 172.21.15.38:0/1198849480 >> 172.21.15.38:0/1198849480 conn(0x7fd6d40f89a0 msgr2=0x7fd6d4106200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:05.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:05.126+0000 7fd6dc30b700 1 -- 172.21.15.38:0/1198849480 shutdown_connections 2024-04-22T21:07:05.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:05.126+0000 7fd6dc30b700 1 -- 172.21.15.38:0/1198849480 wait complete. 2024-04-22T21:07:05.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:05 smithi038 ceph-mon[26374]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:05.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:05 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1198849480' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T21:07:05.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:05 smithi098 ceph-mon[34053]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:05.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:05 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/1198849480' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T21:07:05.751 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-22T21:07:05.751 INFO:tasks.cephadm:Setup complete, yielding 2024-04-22T21:07:05.752 INFO:teuthology.run_tasks:Running task print... 2024-04-22T21:07:05.760 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-04-22T21:07:05.760 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T21:07:05.780 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T21:07:05.780 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-22T21:07:06.050 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:07.206 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:07 smithi038 ceph-mon[26374]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:07.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.425+0000 7f76f9a84700 1 -- 172.21.15.38:0/181694776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40fdf90 msgr2=0x7f76f40fe400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:07.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.425+0000 7f76f9a84700 1 --2- 172.21.15.38:0/181694776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40fdf90 0x7f76f40fe400 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f76e8004940 tx=0x7f76e8009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:07.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.425+0000 7f76f9a84700 1 -- 172.21.15.38:0/181694776 shutdown_connections 2024-04-22T21:07:07.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.425+0000 7f76f9a84700 1 --2- 172.21.15.38:0/181694776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40fdf90 0x7f76f40fe400 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:07.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.425+0000 7f76f9a84700 1 --2- 172.21.15.38:0/181694776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f40fce20 0x7f76f40fd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:07.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.425+0000 7f76f9a84700 1 -- 172.21.15.38:0/181694776 >> 172.21.15.38:0/181694776 conn(0x7f76f40f84e0 msgr2=0x7f76f40fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:07.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.425+0000 7f76f9a84700 1 -- 172.21.15.38:0/181694776 shutdown_connections 2024-04-22T21:07:07.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.425+0000 7f76f9a84700 1 -- 172.21.15.38:0/181694776 wait complete. 2024-04-22T21:07:07.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.425+0000 7f76f9a84700 1 Processor -- start 2024-04-22T21:07:07.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.426+0000 7f76f9a84700 1 -- start start 2024-04-22T21:07:07.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.426+0000 7f76f9a84700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f40fce20 0x7f76f410d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:07.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.426+0000 7f76f9a84700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40fdf90 0x7f76f410de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:07.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.426+0000 7f76f9a84700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f410e490 con 0x7f76f40fdf90 2024-04-22T21:07:07.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.426+0000 7f76f9a84700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f410ece0 con 0x7f76f40fce20 2024-04-22T21:07:07.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.426+0000 7f76f2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40fdf90 0x7f76f410de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:07.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.426+0000 7f76f2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40fdf90 0x7f76f410de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50342/0 (socket says 172.21.15.38:50342) 2024-04-22T21:07:07.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.426+0000 7f76f2ffd700 1 -- 172.21.15.38:0/2061984798 learned_addr learned my addr 172.21.15.38:0/2061984798 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:07.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.426+0000 7f76f37fe700 1 --2- 172.21.15.38:0/2061984798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f40fce20 0x7f76f410d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:07.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.426+0000 7f76f2ffd700 1 -- 172.21.15.38:0/2061984798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f40fce20 msgr2=0x7f76f410d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:07.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.426+0000 7f76f2ffd700 1 --2- 172.21.15.38:0/2061984798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f40fce20 0x7f76f410d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:07.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.426+0000 7f76f2ffd700 1 -- 172.21.15.38:0/2061984798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e8005040 con 0x7f76f40fdf90 2024-04-22T21:07:07.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.426+0000 7f76f37fe700 1 --2- 172.21.15.38:0/2061984798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f40fce20 0x7f76f410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T21:07:07.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.426+0000 7f76f2ffd700 1 --2- 172.21.15.38:0/2061984798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40fdf90 0x7f76f410de90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f76e800b3b0 tx=0x7f76e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:07.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.427+0000 7f76dbfff700 1 -- 172.21.15.38:0/2061984798 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76e800e040 con 0x7f76f40fdf90 2024-04-22T21:07:07.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.427+0000 7f76dbfff700 1 -- 172.21.15.38:0/2061984798 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f76e800bcc0 con 0x7f76f40fdf90 2024-04-22T21:07:07.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.427+0000 7f76f9a84700 1 -- 172.21.15.38:0/2061984798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f410ef60 con 0x7f76f40fdf90 2024-04-22T21:07:07.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.427+0000 7f76dbfff700 1 -- 172.21.15.38:0/2061984798 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76e801c630 con 0x7f76f40fdf90 2024-04-22T21:07:07.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.427+0000 7f76f9a84700 1 -- 172.21.15.38:0/2061984798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f410f4a0 con 0x7f76f40fdf90 2024-04-22T21:07:07.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.428+0000 7f76f9a84700 1 -- 172.21.15.38:0/2061984798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f405f5f0 con 0x7f76f40fdf90 2024-04-22T21:07:07.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.428+0000 7f76dbfff700 1 -- 172.21.15.38:0/2061984798 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f76e8013980 con 0x7f76f40fdf90 2024-04-22T21:07:07.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.429+0000 7f76dbfff700 1 --2- 172.21.15.38:0/2061984798 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f76dc0692a0 0x7f76dc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:07.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.429+0000 7f76dbfff700 1 -- 172.21.15.38:0/2061984798 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f76e8087bd0 con 0x7f76f40fdf90 2024-04-22T21:07:07.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.429+0000 7f76f37fe700 1 --2- 172.21.15.38:0/2061984798 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f76dc0692a0 0x7f76dc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:07.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.429+0000 7f76f37fe700 1 --2- 172.21.15.38:0/2061984798 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f76dc0692a0 0x7f76dc06b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f76e4007fd0 tx=0x7f76e4009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:07.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.433+0000 7f76dbfff700 1 -- 172.21.15.38:0/2061984798 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f76e8057cd0 con 0x7f76f40fdf90 2024-04-22T21:07:07.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:07 smithi098 ceph-mon[34053]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:07.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.627+0000 7f76f9a84700 1 -- 172.21.15.38:0/2061984798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f76f404bed0 con 0x7f76f40fdf90 2024-04-22T21:07:07.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.632+0000 7f76dbfff700 1 -- 172.21.15.38:0/2061984798 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f76e8057860 con 0x7f76f40fdf90 2024-04-22T21:07:07.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.634+0000 7f76f9a84700 1 -- 172.21.15.38:0/2061984798 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f76dc0692a0 msgr2=0x7f76dc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:07.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.634+0000 7f76f9a84700 1 --2- 172.21.15.38:0/2061984798 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f76dc0692a0 0x7f76dc06b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f76e4007fd0 tx=0x7f76e4009040 comp rx=0 tx=0).stop 2024-04-22T21:07:07.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.634+0000 7f76f9a84700 1 -- 172.21.15.38:0/2061984798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40fdf90 msgr2=0x7f76f410de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:07.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.634+0000 7f76f9a84700 1 --2- 172.21.15.38:0/2061984798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40fdf90 0x7f76f410de90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f76e800b3b0 tx=0x7f76e8009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:07.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.634+0000 7f76f9a84700 1 -- 172.21.15.38:0/2061984798 shutdown_connections 2024-04-22T21:07:07.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.634+0000 7f76f9a84700 1 --2- 172.21.15.38:0/2061984798 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f76dc0692a0 0x7f76dc06b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:07.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.634+0000 7f76f9a84700 1 --2- 172.21.15.38:0/2061984798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76f40fce20 0x7f76f410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:07.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.634+0000 7f76f9a84700 1 --2- 172.21.15.38:0/2061984798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40fdf90 0x7f76f410de90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:07.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.634+0000 7f76f9a84700 1 -- 172.21.15.38:0/2061984798 >> 172.21.15.38:0/2061984798 conn(0x7f76f40f84e0 msgr2=0x7f76f40f9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:07.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.635+0000 7f76f9a84700 1 -- 172.21.15.38:0/2061984798 shutdown_connections 2024-04-22T21:07:07.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:07.635+0000 7f76f9a84700 1 -- 172.21.15.38:0/2061984798 wait complete. 2024-04-22T21:07:08.218 INFO:teuthology.run_tasks:Running task print... 2024-04-22T21:07:08.228 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-22T21:07:08.228 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T21:07:08.236 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T21:07:08.236 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-22T21:07:08.507 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:08.888 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:08 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/2061984798' entity='client.admin' 2024-04-22T21:07:08.888 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:08 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:07:08.889 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:08 smithi038 ceph-mon[26374]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:08.889 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:08 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:07:08.889 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:08 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:07:08.889 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:08 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:07:08.889 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:08 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:09.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:08 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/2061984798' entity='client.admin' 2024-04-22T21:07:09.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:08 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:07:09.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:08 smithi098 ceph-mon[34053]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:09.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:08 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:07:09.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:08 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:07:09.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:08 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:07:09.001 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:08 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:09.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.901+0000 7f39d9c3e700 1 -- 172.21.15.38:0/3492208012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39d40fe450 msgr2=0x7f39d40fe860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:09.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.901+0000 7f39d9c3e700 1 --2- 172.21.15.38:0/3492208012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39d40fe450 0x7f39d40fe860 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f39c4004940 tx=0x7f39c4009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:09.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.901+0000 7f39d9c3e700 1 -- 172.21.15.38:0/3492208012 shutdown_connections 2024-04-22T21:07:09.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.901+0000 7f39d9c3e700 1 --2- 172.21.15.38:0/3492208012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39d40feda0 0x7f39d40faee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:09.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.901+0000 7f39d9c3e700 1 --2- 172.21.15.38:0/3492208012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39d40fe450 0x7f39d40fe860 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:09.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.901+0000 7f39d9c3e700 1 -- 172.21.15.38:0/3492208012 >> 172.21.15.38:0/3492208012 conn(0x7f39d40f6bf0 msgr2=0x7f39d40f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:09.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.901+0000 7f39d9c3e700 1 -- 172.21.15.38:0/3492208012 shutdown_connections 2024-04-22T21:07:09.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.901+0000 7f39d9c3e700 1 -- 172.21.15.38:0/3492208012 wait complete. 2024-04-22T21:07:09.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.902+0000 7f39d9c3e700 1 Processor -- start 2024-04-22T21:07:09.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.902+0000 7f39d9c3e700 1 -- start start 2024-04-22T21:07:09.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.902+0000 7f39d9c3e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39d40fe450 0x7f39d410fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:09.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.902+0000 7f39d9c3e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39d40feda0 0x7f39d41100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:09.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.902+0000 7f39d9c3e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d41106a0 con 0x7f39d40feda0 2024-04-22T21:07:09.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.902+0000 7f39d9c3e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d4110810 con 0x7f39d40fe450 2024-04-22T21:07:09.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.902+0000 7f39d2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39d40feda0 0x7f39d41100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:09.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.902+0000 7f39d2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39d40feda0 0x7f39d41100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50368/0 (socket says 172.21.15.38:50368) 2024-04-22T21:07:09.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.902+0000 7f39d2ffd700 1 -- 172.21.15.38:0/3892358129 learned_addr learned my addr 172.21.15.38:0/3892358129 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:09.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.902+0000 7f39d37fe700 1 --2- 172.21.15.38:0/3892358129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39d40fe450 0x7f39d410fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:09.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.902+0000 7f39d2ffd700 1 -- 172.21.15.38:0/3892358129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39d40fe450 msgr2=0x7f39d410fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:09.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.902+0000 7f39d2ffd700 1 --2- 172.21.15.38:0/3892358129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39d40fe450 0x7f39d410fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:09.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.902+0000 7f39d2ffd700 1 -- 172.21.15.38:0/3892358129 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c4005040 con 0x7f39d40feda0 2024-04-22T21:07:09.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.903+0000 7f39d2ffd700 1 --2- 172.21.15.38:0/3892358129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39d40feda0 0x7f39d41100a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f39c800c8c0 tx=0x7f39c80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:09.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.903+0000 7f39bbfff700 1 -- 172.21.15.38:0/3892358129 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39c800a760 con 0x7f39d40feda0 2024-04-22T21:07:09.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.903+0000 7f39d9c3e700 1 -- 172.21.15.38:0/3892358129 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d4115270 con 0x7f39d40feda0 2024-04-22T21:07:09.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.903+0000 7f39bbfff700 1 -- 172.21.15.38:0/3892358129 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39c800ad90 con 0x7f39d40feda0 2024-04-22T21:07:09.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.903+0000 7f39bbfff700 1 -- 172.21.15.38:0/3892358129 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39c80113a0 con 0x7f39d40feda0 2024-04-22T21:07:09.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.903+0000 7f39d9c3e700 1 -- 172.21.15.38:0/3892358129 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d4115830 con 0x7f39d40feda0 2024-04-22T21:07:09.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.904+0000 7f39d9c3e700 1 -- 172.21.15.38:0/3892358129 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d4101b90 con 0x7f39d40feda0 2024-04-22T21:07:09.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.904+0000 7f39bbfff700 1 -- 172.21.15.38:0/3892358129 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f39c8012400 con 0x7f39d40feda0 2024-04-22T21:07:09.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.905+0000 7f39bbfff700 1 --2- 172.21.15.38:0/3892358129 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f39bc0692a0 0x7f39bc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:09.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.905+0000 7f39bbfff700 1 -- 172.21.15.38:0/3892358129 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f39c8086e70 con 0x7f39d40feda0 2024-04-22T21:07:09.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.905+0000 7f39d37fe700 1 --2- 172.21.15.38:0/3892358129 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f39bc0692a0 0x7f39bc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:09.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.905+0000 7f39d37fe700 1 --2- 172.21.15.38:0/3892358129 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f39bc0692a0 0x7f39bc06b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f39c400b970 tx=0x7f39c4009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:09.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:09.909+0000 7f39bbfff700 1 -- 172.21.15.38:0/3892358129 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f39c8056f70 con 0x7f39d40feda0 2024-04-22T21:07:10.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:10.120+0000 7f39d9c3e700 1 -- 172.21.15.38:0/3892358129 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f39d402b310 con 0x7f39bc0692a0 2024-04-22T21:07:10.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:10.121+0000 7f39bbfff700 1 -- 172.21.15.38:0/3892358129 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f39d402b310 con 0x7f39bc0692a0 2024-04-22T21:07:10.117 INFO:teuthology.orchestra.run.smithi038.stdout:Backend: cephadm 2024-04-22T21:07:10.117 INFO:teuthology.orchestra.run.smithi038.stdout:Available: Yes 2024-04-22T21:07:10.118 INFO:teuthology.orchestra.run.smithi038.stdout:Paused: No 2024-04-22T21:07:10.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:10.123+0000 7f39d9c3e700 1 -- 172.21.15.38:0/3892358129 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f39bc0692a0 msgr2=0x7f39bc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:10.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:10.123+0000 7f39d9c3e700 1 --2- 172.21.15.38:0/3892358129 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f39bc0692a0 0x7f39bc06b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f39c400b970 tx=0x7f39c4009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:10.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:10.123+0000 7f39d9c3e700 1 -- 172.21.15.38:0/3892358129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39d40feda0 msgr2=0x7f39d41100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:10.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:10.123+0000 7f39d9c3e700 1 --2- 172.21.15.38:0/3892358129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39d40feda0 0x7f39d41100a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f39c800c8c0 tx=0x7f39c80044c0 comp rx=0 tx=0).stop 2024-04-22T21:07:10.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:10.123+0000 7f39d9c3e700 1 -- 172.21.15.38:0/3892358129 shutdown_connections 2024-04-22T21:07:10.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:10.123+0000 7f39d9c3e700 1 --2- 172.21.15.38:0/3892358129 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f39bc0692a0 0x7f39bc06b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:10.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:10.123+0000 7f39d9c3e700 1 --2- 172.21.15.38:0/3892358129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39d40fe450 0x7f39d410fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:10.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:10.123+0000 7f39d9c3e700 1 --2- 172.21.15.38:0/3892358129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39d40feda0 0x7f39d41100a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:10.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:10.123+0000 7f39d9c3e700 1 -- 172.21.15.38:0/3892358129 >> 172.21.15.38:0/3892358129 conn(0x7f39d40f6bf0 msgr2=0x7f39d40f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:10.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:10.123+0000 7f39d9c3e700 1 -- 172.21.15.38:0/3892358129 shutdown_connections 2024-04-22T21:07:10.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:10.123+0000 7f39d9c3e700 1 -- 172.21.15.38:0/3892358129 wait complete. 2024-04-22T21:07:10.717 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-22T21:07:10.979 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:11.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:11 smithi038 ceph-mon[26374]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:11.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:11 smithi038 ceph-mon[26374]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:07:11.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:11 smithi098 ceph-mon[34053]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:11.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:11 smithi098 ceph-mon[34053]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:07:12.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.542+0000 7fc46ef12700 1 -- 172.21.15.38:0/1502540537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4680fcff0 msgr2=0x7fc4680fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:12.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.542+0000 7fc46ef12700 1 --2- 172.21.15.38:0/1502540537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4680fcff0 0x7fc4680fd460 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc454004880 tx=0x7fc454009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:12.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.542+0000 7fc46ef12700 1 -- 172.21.15.38:0/1502540537 shutdown_connections 2024-04-22T21:07:12.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.542+0000 7fc46ef12700 1 --2- 172.21.15.38:0/1502540537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4680fcff0 0x7fc4680fd460 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:12.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.542+0000 7fc46ef12700 1 --2- 172.21.15.38:0/1502540537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4680fbd40 0x7fc4680fc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:12.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.542+0000 7fc46ef12700 1 -- 172.21.15.38:0/1502540537 >> 172.21.15.38:0/1502540537 conn(0x7fc4680f74c0 msgr2=0x7fc4680f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:12.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.542+0000 7fc46ef12700 1 -- 172.21.15.38:0/1502540537 shutdown_connections 2024-04-22T21:07:12.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.542+0000 7fc46ef12700 1 -- 172.21.15.38:0/1502540537 wait complete. 2024-04-22T21:07:12.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.543+0000 7fc46ef12700 1 Processor -- start 2024-04-22T21:07:12.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.543+0000 7fc46ef12700 1 -- start start 2024-04-22T21:07:12.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.543+0000 7fc46ef12700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4680fbd40 0x7fc46810da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:12.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.543+0000 7fc46ef12700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4680fcff0 0x7fc46810df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:12.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.543+0000 7fc46ef12700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46810e550 con 0x7fc4680fcff0 2024-04-22T21:07:12.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.543+0000 7fc46ef12700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46810eda0 con 0x7fc4680fbd40 2024-04-22T21:07:12.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.543+0000 7fc467fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4680fcff0 0x7fc46810df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:12.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.543+0000 7fc46ccae700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4680fbd40 0x7fc46810da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:12.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.543+0000 7fc467fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4680fcff0 0x7fc46810df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50386/0 (socket says 172.21.15.38:50386) 2024-04-22T21:07:12.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.543+0000 7fc46ccae700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4680fbd40 0x7fc46810da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.38:41006/0 (socket says 172.21.15.38:41006) 2024-04-22T21:07:12.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.543+0000 7fc467fff700 1 -- 172.21.15.38:0/2704379202 learned_addr learned my addr 172.21.15.38:0/2704379202 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:12.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.544+0000 7fc467fff700 1 -- 172.21.15.38:0/2704379202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4680fbd40 msgr2=0x7fc46810da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:12.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.544+0000 7fc467fff700 1 --2- 172.21.15.38:0/2704379202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4680fbd40 0x7fc46810da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:12.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.544+0000 7fc467fff700 1 -- 172.21.15.38:0/2704379202 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc454005040 con 0x7fc4680fcff0 2024-04-22T21:07:12.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.544+0000 7fc46ccae700 1 --2- 172.21.15.38:0/2704379202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4680fbd40 0x7fc46810da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T21:07:12.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.544+0000 7fc467fff700 1 --2- 172.21.15.38:0/2704379202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4680fcff0 0x7fc46810df50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc454008d00 tx=0x7fc454009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:12.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.544+0000 7fc4657fa700 1 -- 172.21.15.38:0/2704379202 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc45400e030 con 0x7fc4680fcff0 2024-04-22T21:07:12.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.544+0000 7fc46ef12700 1 -- 172.21.15.38:0/2704379202 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc46810f020 con 0x7fc4680fcff0 2024-04-22T21:07:12.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.544+0000 7fc4657fa700 1 -- 172.21.15.38:0/2704379202 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc454013e60 con 0x7fc4680fcff0 2024-04-22T21:07:12.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.544+0000 7fc4657fa700 1 -- 172.21.15.38:0/2704379202 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4540123c0 con 0x7fc4680fcff0 2024-04-22T21:07:12.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.544+0000 7fc46ef12700 1 -- 172.21.15.38:0/2704379202 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc46810f560 con 0x7fc4680fcff0 2024-04-22T21:07:12.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.545+0000 7fc46ef12700 1 -- 172.21.15.38:0/2704379202 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc468101b80 con 0x7fc4680fcff0 2024-04-22T21:07:12.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.545+0000 7fc4657fa700 1 -- 172.21.15.38:0/2704379202 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fc45401c400 con 0x7fc4680fcff0 2024-04-22T21:07:12.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.546+0000 7fc4657fa700 1 --2- 172.21.15.38:0/2704379202 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc4580692b0 0x7fc45806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:12.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.546+0000 7fc4657fa700 1 -- 172.21.15.38:0/2704379202 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc454088890 con 0x7fc4680fcff0 2024-04-22T21:07:12.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.546+0000 7fc46ccae700 1 --2- 172.21.15.38:0/2704379202 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc4580692b0 0x7fc45806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:12.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.546+0000 7fc46ccae700 1 --2- 172.21.15.38:0/2704379202 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc4580692b0 0x7fc45806b760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc45c009b80 tx=0x7fc45c00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:12.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.551+0000 7fc4657fa700 1 -- 172.21.15.38:0/2704379202 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fc454058a50 con 0x7fc4680fcff0 2024-04-22T21:07:12.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.756+0000 7fc46ef12700 1 -- 172.21.15.38:0/2704379202 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc46810f780 con 0x7fc4580692b0 2024-04-22T21:07:12.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.766+0000 7fc4657fa700 1 -- 172.21.15.38:0/2704379202 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2682 (secure 0 0 0) 0x7fc46810f780 con 0x7fc4580692b0 2024-04-22T21:07:12.762 INFO:teuthology.orchestra.run.smithi038.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T21:07:12.762 INFO:teuthology.orchestra.run.smithi038.stdout:alertmanager.smithi038 smithi038 *:9093,9094 running (3m) 2m ago 5m 25.6M - 0.25.0 c8568f914cd2 28f5f3f389c1 2024-04-22T21:07:12.762 INFO:teuthology.orchestra.run.smithi038.stdout:crash.smithi038 smithi038 running (5m) 2m ago 5m 6665k - 17.2.7-884-g70d0cb4b cafd8ddac8fa ff0889f35e01 2024-04-22T21:07:12.762 INFO:teuthology.orchestra.run.smithi038.stdout:crash.smithi098 smithi098 running (4m) 63s ago 4m 6656k - 17.2.7-884-g70d0cb4b cafd8ddac8fa ced65820ce17 2024-04-22T21:07:12.762 INFO:teuthology.orchestra.run.smithi038.stdout:grafana.smithi038 smithi038 *:3000 running (3m) 2m ago 4m 78.3M - 9.4.7 954c08fa6188 849b80d16433 2024-04-22T21:07:12.762 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.smithi038.lxfgut smithi038 *:9283 running (6m) 2m ago 6m 469M - 17.2.7-884-g70d0cb4b cafd8ddac8fa ee36b4dede58 2024-04-22T21:07:12.762 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.smithi098.azqohf smithi098 *:8443,9283 running (4m) 63s ago 4m 434M - 17.2.7-884-g70d0cb4b cafd8ddac8fa c3fae1c785a2 2024-04-22T21:07:12.763 INFO:teuthology.orchestra.run.smithi038.stdout:mon.smithi038 smithi038 running (6m) 2m ago 6m 40.6M 2048M 17.2.7-884-g70d0cb4b cafd8ddac8fa 2137696f8742 2024-04-22T21:07:12.763 INFO:teuthology.orchestra.run.smithi038.stdout:mon.smithi098 smithi098 running (4m) 63s ago 4m 38.1M 2048M 17.2.7-884-g70d0cb4b cafd8ddac8fa e996b8c559ed 2024-04-22T21:07:12.763 INFO:teuthology.orchestra.run.smithi038.stdout:node-exporter.smithi038 smithi038 *:9100 running (5m) 2m ago 5m 15.5M - 1.5.0 0da6a335fe13 347a9e37455f 2024-04-22T21:07:12.763 INFO:teuthology.orchestra.run.smithi038.stdout:node-exporter.smithi098 smithi098 *:9100 running (4m) 63s ago 4m 16.2M - 1.5.0 0da6a335fe13 5274c36423cc 2024-04-22T21:07:12.763 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 smithi038 running (3m) 2m ago 3m 34.4M 4289M 17.2.7-884-g70d0cb4b cafd8ddac8fa 160468eff0dc 2024-04-22T21:07:12.763 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 smithi038 running (2m) 2m ago 2m 38.3M 4289M 17.2.7-884-g70d0cb4b cafd8ddac8fa d79042851a05 2024-04-22T21:07:12.763 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2 smithi038 running (2m) 2m ago 2m 13.2M 4289M 17.2.7-884-g70d0cb4b cafd8ddac8fa 8aaa1e69183c 2024-04-22T21:07:12.763 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3 smithi098 running (2m) 63s ago 2m 40.0M 5312M 17.2.7-884-g70d0cb4b cafd8ddac8fa a59b0f44b178 2024-04-22T21:07:12.763 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 smithi098 running (94s) 63s ago 94s 36.4M 5312M 17.2.7-884-g70d0cb4b cafd8ddac8fa 848d77af5440 2024-04-22T21:07:12.763 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 smithi098 running (69s) 63s ago 68s 13.3M 5312M 17.2.7-884-g70d0cb4b cafd8ddac8fa ef4d55a626c6 2024-04-22T21:07:12.763 INFO:teuthology.orchestra.run.smithi038.stdout:prometheus.smithi038 smithi038 *:9095 running (3m) 2m ago 4m 35.1M - 2.43.0 a07b618ecd1d 5cc3d358fbbd 2024-04-22T21:07:12.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.768+0000 7fc46ef12700 1 -- 172.21.15.38:0/2704379202 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc4580692b0 msgr2=0x7fc45806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:12.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.768+0000 7fc46ef12700 1 --2- 172.21.15.38:0/2704379202 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc4580692b0 0x7fc45806b760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc45c009b80 tx=0x7fc45c00b040 comp rx=0 tx=0).stop 2024-04-22T21:07:12.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.768+0000 7fc46ef12700 1 -- 172.21.15.38:0/2704379202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4680fcff0 msgr2=0x7fc46810df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:12.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.768+0000 7fc46ef12700 1 --2- 172.21.15.38:0/2704379202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4680fcff0 0x7fc46810df50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc454008d00 tx=0x7fc454009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:12.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.768+0000 7fc46ef12700 1 -- 172.21.15.38:0/2704379202 shutdown_connections 2024-04-22T21:07:12.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.768+0000 7fc46ef12700 1 --2- 172.21.15.38:0/2704379202 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc4580692b0 0x7fc45806b760 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:12.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.768+0000 7fc46ef12700 1 --2- 172.21.15.38:0/2704379202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4680fbd40 0x7fc46810da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:12.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.768+0000 7fc46ef12700 1 --2- 172.21.15.38:0/2704379202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4680fcff0 0x7fc46810df50 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:12.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.768+0000 7fc46ef12700 1 -- 172.21.15.38:0/2704379202 >> 172.21.15.38:0/2704379202 conn(0x7fc4680f74c0 msgr2=0x7fc468065be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:12.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.769+0000 7fc46ef12700 1 -- 172.21.15.38:0/2704379202 shutdown_connections 2024-04-22T21:07:12.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:12.769+0000 7fc46ef12700 1 -- 172.21.15.38:0/2704379202 wait complete. 2024-04-22T21:07:13.322 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-22T21:07:13.345 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:13 smithi038 ceph-mon[26374]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:13.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:13 smithi098 ceph-mon[34053]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:13.593 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:14.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:14 smithi038 ceph-mon[26374]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:07:14.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:14 smithi098 ceph-mon[34053]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:07:15.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.232+0000 7fee51f51700 1 -- 172.21.15.38:0/2437031580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c066f00 msgr2=0x7fee4c0ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:15.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.232+0000 7fee51f51700 1 --2- 172.21.15.38:0/2437031580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c066f00 0x7fee4c0ffa40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009ce0 tx=0x7fee3c004d10 comp rx=0 tx=0).stop 2024-04-22T21:07:15.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.232+0000 7fee51f51700 1 -- 172.21.15.38:0/2437031580 shutdown_connections 2024-04-22T21:07:15.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.232+0000 7fee51f51700 1 --2- 172.21.15.38:0/2437031580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee4c0fff80 0x7fee4c102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:15.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.232+0000 7fee51f51700 1 --2- 172.21.15.38:0/2437031580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c066f00 0x7fee4c0ffa40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:15.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.232+0000 7fee51f51700 1 -- 172.21.15.38:0/2437031580 >> 172.21.15.38:0/2437031580 conn(0x7fee4c0f7450 msgr2=0x7fee4c0f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:15.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.232+0000 7fee51f51700 1 -- 172.21.15.38:0/2437031580 shutdown_connections 2024-04-22T21:07:15.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.233+0000 7fee51f51700 1 -- 172.21.15.38:0/2437031580 wait complete. 2024-04-22T21:07:15.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.233+0000 7fee51f51700 1 Processor -- start 2024-04-22T21:07:15.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.233+0000 7fee51f51700 1 -- start start 2024-04-22T21:07:15.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.233+0000 7fee51f51700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee4c066f00 0x7fee4c10de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:15.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.233+0000 7fee51f51700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c0fff80 0x7fee4c10e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:15.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.234+0000 7fee51f51700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c10e990 con 0x7fee4c0fff80 2024-04-22T21:07:15.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.234+0000 7fee51f51700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c10eb00 con 0x7fee4c066f00 2024-04-22T21:07:15.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.234+0000 7fee4affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c0fff80 0x7fee4c10e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:15.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.234+0000 7fee4affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c0fff80 0x7fee4c10e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34864/0 (socket says 172.21.15.38:34864) 2024-04-22T21:07:15.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.234+0000 7fee4b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee4c066f00 0x7fee4c10de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:15.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.234+0000 7fee4affd700 1 -- 172.21.15.38:0/4018566956 learned_addr learned my addr 172.21.15.38:0/4018566956 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:15.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.234+0000 7fee4affd700 1 -- 172.21.15.38:0/4018566956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee4c066f00 msgr2=0x7fee4c10de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:15.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.234+0000 7fee4affd700 1 --2- 172.21.15.38:0/4018566956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee4c066f00 0x7fee4c10de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:15.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.234+0000 7fee4affd700 1 -- 172.21.15.38:0/4018566956 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3c009940 con 0x7fee4c0fff80 2024-04-22T21:07:15.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.234+0000 7fee4affd700 1 --2- 172.21.15.38:0/4018566956 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c0fff80 0x7fee4c10e390 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fee34002a50 tx=0x7fee340046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:15.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.234+0000 7fee43fff700 1 -- 172.21.15.38:0/4018566956 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee34007cc0 con 0x7fee4c0fff80 2024-04-22T21:07:15.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.234+0000 7fee43fff700 1 -- 172.21.15.38:0/4018566956 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee3400f040 con 0x7fee4c0fff80 2024-04-22T21:07:15.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.234+0000 7fee51f51700 1 -- 172.21.15.38:0/4018566956 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee4c113560 con 0x7fee4c0fff80 2024-04-22T21:07:15.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.234+0000 7fee43fff700 1 -- 172.21.15.38:0/4018566956 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee3400da70 con 0x7fee4c0fff80 2024-04-22T21:07:15.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.234+0000 7fee51f51700 1 -- 172.21.15.38:0/4018566956 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4c113b20 con 0x7fee4c0fff80 2024-04-22T21:07:15.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.235+0000 7fee51f51700 1 -- 172.21.15.38:0/4018566956 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4c04c710 con 0x7fee4c0fff80 2024-04-22T21:07:15.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.236+0000 7fee43fff700 1 -- 172.21.15.38:0/4018566956 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fee3400dc10 con 0x7fee4c0fff80 2024-04-22T21:07:15.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.236+0000 7fee43fff700 1 --2- 172.21.15.38:0/4018566956 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fee380692a0 0x7fee3806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:15.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.236+0000 7fee4b7fe700 1 --2- 172.21.15.38:0/4018566956 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fee380692a0 0x7fee3806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:15.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.236+0000 7fee43fff700 1 -- 172.21.15.38:0/4018566956 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fee340864e0 con 0x7fee4c0fff80 2024-04-22T21:07:15.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.237+0000 7fee4b7fe700 1 --2- 172.21.15.38:0/4018566956 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fee380692a0 0x7fee3806b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009470 tx=0x7fee3c004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:15.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.241+0000 7fee43fff700 1 -- 172.21.15.38:0/4018566956 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fee34056560 con 0x7fee4c0fff80 2024-04-22T21:07:15.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:15 smithi038 ceph-mon[26374]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:15.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.441+0000 7fee51f51700 1 -- 172.21.15.38:0/4018566956 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fee4c101b50 con 0x7fee380692a0 2024-04-22T21:07:15.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.445+0000 7fee43fff700 1 -- 172.21.15.38:0/4018566956 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1161 (secure 0 0 0) 0x7fee4c101b50 con 0x7fee380692a0 2024-04-22T21:07:15.441 INFO:teuthology.orchestra.run.smithi038.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-22T21:07:15.441 INFO:teuthology.orchestra.run.smithi038.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-04-22T21:07:15.441 INFO:teuthology.orchestra.run.smithi038.stdout:crash 2/2 2m ago 6m * 2024-04-22T21:07:15.441 INFO:teuthology.orchestra.run.smithi038.stdout:grafana ?:3000 1/1 2m ago 5m count:1 2024-04-22T21:07:15.441 INFO:teuthology.orchestra.run.smithi038.stdout:mgr 2/2 2m ago 6m count:2 2024-04-22T21:07:15.442 INFO:teuthology.orchestra.run.smithi038.stdout:mon 2/2 2m ago 5m smithi038:172.21.15.38=smithi038;smithi098:172.21.15.98=smithi098;count:2 2024-04-22T21:07:15.442 INFO:teuthology.orchestra.run.smithi038.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2024-04-22T21:07:15.442 INFO:teuthology.orchestra.run.smithi038.stdout:osd 6 2m ago - 2024-04-22T21:07:15.442 INFO:teuthology.orchestra.run.smithi038.stdout:prometheus ?:9095 1/1 2m ago 5m count:1 2024-04-22T21:07:15.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.447+0000 7fee51f51700 1 -- 172.21.15.38:0/4018566956 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fee380692a0 msgr2=0x7fee3806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:15.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.447+0000 7fee51f51700 1 --2- 172.21.15.38:0/4018566956 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fee380692a0 0x7fee3806b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009470 tx=0x7fee3c004d10 comp rx=0 tx=0).stop 2024-04-22T21:07:15.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.447+0000 7fee51f51700 1 -- 172.21.15.38:0/4018566956 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c0fff80 msgr2=0x7fee4c10e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:15.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.447+0000 7fee51f51700 1 --2- 172.21.15.38:0/4018566956 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c0fff80 0x7fee4c10e390 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fee34002a50 tx=0x7fee340046b0 comp rx=0 tx=0).stop 2024-04-22T21:07:15.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.447+0000 7fee51f51700 1 -- 172.21.15.38:0/4018566956 shutdown_connections 2024-04-22T21:07:15.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.447+0000 7fee51f51700 1 --2- 172.21.15.38:0/4018566956 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fee380692a0 0x7fee3806b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:15.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.447+0000 7fee51f51700 1 --2- 172.21.15.38:0/4018566956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee4c066f00 0x7fee4c10de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:15.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.447+0000 7fee51f51700 1 --2- 172.21.15.38:0/4018566956 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c0fff80 0x7fee4c10e390 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:15.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.447+0000 7fee51f51700 1 -- 172.21.15.38:0/4018566956 >> 172.21.15.38:0/4018566956 conn(0x7fee4c0f7450 msgr2=0x7fee4c0f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:15.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.447+0000 7fee51f51700 1 -- 172.21.15.38:0/4018566956 shutdown_connections 2024-04-22T21:07:15.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:15.447+0000 7fee51f51700 1 -- 172.21.15.38:0/4018566956 wait complete. 2024-04-22T21:07:15.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:15 smithi098 ceph-mon[34053]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:16.040 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-22T21:07:16.349 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:17.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:17 smithi038 ceph-mon[26374]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:07:17.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:17 smithi038 ceph-mon[26374]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:17.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:17 smithi098 ceph-mon[34053]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:07:17.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:17 smithi098 ceph-mon[34053]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:17.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.951+0000 7f693dba1700 1 -- 172.21.15.38:0/4204589176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69380ffd40 msgr2=0x7f6938102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:17.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.951+0000 7f693dba1700 1 --2- 172.21.15.38:0/4204589176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69380ffd40 0x7f6938102160 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6924004940 tx=0x7f6924009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:17.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.951+0000 7f693dba1700 1 -- 172.21.15.38:0/4204589176 shutdown_connections 2024-04-22T21:07:17.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.951+0000 7f693dba1700 1 --2- 172.21.15.38:0/4204589176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69380ffd40 0x7f6938102160 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:17.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.951+0000 7f693dba1700 1 --2- 172.21.15.38:0/4204589176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69380662f0 0x7f69380ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:17.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.951+0000 7f693dba1700 1 -- 172.21.15.38:0/4204589176 >> 172.21.15.38:0/4204589176 conn(0x7f69380f7270 msgr2=0x7f69380f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:17.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.951+0000 7f693dba1700 1 -- 172.21.15.38:0/4204589176 shutdown_connections 2024-04-22T21:07:17.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.951+0000 7f693dba1700 1 -- 172.21.15.38:0/4204589176 wait complete. 2024-04-22T21:07:17.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.952+0000 7f693dba1700 1 Processor -- start 2024-04-22T21:07:17.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.952+0000 7f693dba1700 1 -- start start 2024-04-22T21:07:17.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.952+0000 7f693dba1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69380662f0 0x7f693810d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:17.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.952+0000 7f693dba1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69380ffd40 0x7f693810de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:17.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.952+0000 7f693dba1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693810e500 con 0x7f69380ffd40 2024-04-22T21:07:17.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.952+0000 7f693dba1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693810ec50 con 0x7f69380662f0 2024-04-22T21:07:17.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.953+0000 7f6937fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69380ffd40 0x7f693810de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:17.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.953+0000 7f6937fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69380ffd40 0x7f693810de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34878/0 (socket says 172.21.15.38:34878) 2024-04-22T21:07:17.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.953+0000 7f6937fff700 1 -- 172.21.15.38:0/2065804983 learned_addr learned my addr 172.21.15.38:0/2065804983 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:17.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.953+0000 7f693cb9f700 1 --2- 172.21.15.38:0/2065804983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69380662f0 0x7f693810d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:17.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.953+0000 7f6937fff700 1 -- 172.21.15.38:0/2065804983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69380662f0 msgr2=0x7f693810d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:17.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.953+0000 7f6937fff700 1 --2- 172.21.15.38:0/2065804983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69380662f0 0x7f693810d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:17.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.953+0000 7f6937fff700 1 -- 172.21.15.38:0/2065804983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6924005040 con 0x7f69380ffd40 2024-04-22T21:07:17.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.953+0000 7f6937fff700 1 --2- 172.21.15.38:0/2065804983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69380ffd40 0x7f693810de70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f692400b970 tx=0x7f6924009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:17.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.953+0000 7f69357fa700 1 -- 172.21.15.38:0/2065804983 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f692400e040 con 0x7f69380ffd40 2024-04-22T21:07:17.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.953+0000 7f693dba1700 1 -- 172.21.15.38:0/2065804983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693810eed0 con 0x7f69380ffd40 2024-04-22T21:07:17.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.953+0000 7f69357fa700 1 -- 172.21.15.38:0/2065804983 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f692400bcc0 con 0x7f69380ffd40 2024-04-22T21:07:17.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.953+0000 7f69357fa700 1 -- 172.21.15.38:0/2065804983 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6924013660 con 0x7f69380ffd40 2024-04-22T21:07:17.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.953+0000 7f693dba1700 1 -- 172.21.15.38:0/2065804983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693810f490 con 0x7f69380ffd40 2024-04-22T21:07:17.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.954+0000 7f693dba1700 1 -- 172.21.15.38:0/2065804983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693805f5f0 con 0x7f69380ffd40 2024-04-22T21:07:17.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.955+0000 7f69357fa700 1 -- 172.21.15.38:0/2065804983 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f692401ca00 con 0x7f69380ffd40 2024-04-22T21:07:17.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.955+0000 7f69357fa700 1 --2- 172.21.15.38:0/2065804983 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f69280692a0 0x7f692806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:17.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.955+0000 7f69357fa700 1 -- 172.21.15.38:0/2065804983 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6924087b80 con 0x7f69380ffd40 2024-04-22T21:07:17.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.955+0000 7f693cb9f700 1 --2- 172.21.15.38:0/2065804983 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f69280692a0 0x7f692806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:17.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.956+0000 7f693cb9f700 1 --2- 172.21.15.38:0/2065804983 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f69280692a0 0x7f692806b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f692c006fd0 tx=0x7f692c008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:17.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:17.960+0000 7f69357fa700 1 -- 172.21.15.38:0/2065804983 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6924057d30 con 0x7f69380ffd40 2024-04-22T21:07:18.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:18.162+0000 7f693dba1700 1 -- 172.21.15.38:0/2065804983 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f693810f720 con 0x7f69280692a0 2024-04-22T21:07:18.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:18.164+0000 7f69357fa700 1 -- 172.21.15.38:0/2065804983 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+145 (secure 0 0 0) 0x7f693810f720 con 0x7f69280692a0 2024-04-22T21:07:18.160 INFO:teuthology.orchestra.run.smithi038.stdout:HOST ADDR LABELS STATUS 2024-04-22T21:07:18.160 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 172.21.15.38 2024-04-22T21:07:18.160 INFO:teuthology.orchestra.run.smithi038.stdout:smithi098 172.21.15.98 2024-04-22T21:07:18.160 INFO:teuthology.orchestra.run.smithi038.stdout:2 hosts in cluster 2024-04-22T21:07:18.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:18.166+0000 7f693dba1700 1 -- 172.21.15.38:0/2065804983 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f69280692a0 msgr2=0x7f692806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:18.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:18.166+0000 7f693dba1700 1 --2- 172.21.15.38:0/2065804983 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f69280692a0 0x7f692806b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f692c006fd0 tx=0x7f692c008040 comp rx=0 tx=0).stop 2024-04-22T21:07:18.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:18.166+0000 7f693dba1700 1 -- 172.21.15.38:0/2065804983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69380ffd40 msgr2=0x7f693810de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:18.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:18.166+0000 7f693dba1700 1 --2- 172.21.15.38:0/2065804983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69380ffd40 0x7f693810de70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f692400b970 tx=0x7f6924009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:18.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:18.166+0000 7f693dba1700 1 -- 172.21.15.38:0/2065804983 shutdown_connections 2024-04-22T21:07:18.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:18.166+0000 7f693dba1700 1 --2- 172.21.15.38:0/2065804983 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f69280692a0 0x7f692806b750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:18.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:18.166+0000 7f693dba1700 1 --2- 172.21.15.38:0/2065804983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69380662f0 0x7f693810d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:18.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:18.166+0000 7f693dba1700 1 --2- 172.21.15.38:0/2065804983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69380ffd40 0x7f693810de70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:18.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:18.166+0000 7f693dba1700 1 -- 172.21.15.38:0/2065804983 >> 172.21.15.38:0/2065804983 conn(0x7f69380f7270 msgr2=0x7f69380f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:18.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:18.166+0000 7f693dba1700 1 -- 172.21.15.38:0/2065804983 shutdown_connections 2024-04-22T21:07:18.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:18.166+0000 7f693dba1700 1 -- 172.21.15.38:0/2065804983 wait complete. 2024-04-22T21:07:18.899 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-22T21:07:19.168 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:19.473 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:19 smithi038 ceph-mon[26374]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:19.473 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:19 smithi038 ceph-mon[26374]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:07:19.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:19 smithi098 ceph-mon[34053]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:19.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:19 smithi098 ceph-mon[34053]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:07:20.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.695+0000 7f555fcaa700 1 -- 172.21.15.38:0/957035690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55580fdff0 msgr2=0x7f55580fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:20.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.695+0000 7f555fcaa700 1 --2- 172.21.15.38:0/957035690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55580fdff0 0x7f55580fe460 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5548004940 tx=0x7f5548009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:20.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.695+0000 7f555fcaa700 1 -- 172.21.15.38:0/957035690 shutdown_connections 2024-04-22T21:07:20.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.695+0000 7f555fcaa700 1 --2- 172.21.15.38:0/957035690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55580fdff0 0x7f55580fe460 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:20.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.695+0000 7f555fcaa700 1 --2- 172.21.15.38:0/957035690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55580fce80 0x7f55580fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:20.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.695+0000 7f555fcaa700 1 -- 172.21.15.38:0/957035690 >> 172.21.15.38:0/957035690 conn(0x7f55580f8580 msgr2=0x7f55580fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:20.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.696+0000 7f555fcaa700 1 -- 172.21.15.38:0/957035690 shutdown_connections 2024-04-22T21:07:20.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.696+0000 7f555fcaa700 1 -- 172.21.15.38:0/957035690 wait complete. 2024-04-22T21:07:20.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.696+0000 7f555fcaa700 1 Processor -- start 2024-04-22T21:07:20.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.696+0000 7f555fcaa700 1 -- start start 2024-04-22T21:07:20.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.696+0000 7f555fcaa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55580fce80 0x7f555810d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:20.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.696+0000 7f555fcaa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55580fdff0 0x7f555810de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:20.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.696+0000 7f555fcaa700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555810e4f0 con 0x7f55580fdff0 2024-04-22T21:07:20.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.696+0000 7f555fcaa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5558112e90 con 0x7f55580fce80 2024-04-22T21:07:20.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.696+0000 7f555d245700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55580fdff0 0x7f555810de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:20.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.696+0000 7f555d245700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55580fdff0 0x7f555810de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34890/0 (socket says 172.21.15.38:34890) 2024-04-22T21:07:20.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.696+0000 7f555d245700 1 -- 172.21.15.38:0/2445963735 learned_addr learned my addr 172.21.15.38:0/2445963735 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:20.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.696+0000 7f555da46700 1 --2- 172.21.15.38:0/2445963735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55580fce80 0x7f555810d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:20.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.697+0000 7f555d245700 1 -- 172.21.15.38:0/2445963735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55580fce80 msgr2=0x7f555810d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:20.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.697+0000 7f555d245700 1 --2- 172.21.15.38:0/2445963735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55580fce80 0x7f555810d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:20.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.697+0000 7f555d245700 1 -- 172.21.15.38:0/2445963735 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5548005040 con 0x7f55580fdff0 2024-04-22T21:07:20.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.697+0000 7f555d245700 1 --2- 172.21.15.38:0/2445963735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55580fdff0 0x7f555810de60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5548009150 tx=0x7f5548009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:20.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.697+0000 7f554e7fc700 1 -- 172.21.15.38:0/2445963735 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f554800e040 con 0x7f55580fdff0 2024-04-22T21:07:20.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.697+0000 7f555fcaa700 1 -- 172.21.15.38:0/2445963735 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5558113090 con 0x7f55580fdff0 2024-04-22T21:07:20.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.697+0000 7f554e7fc700 1 -- 172.21.15.38:0/2445963735 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f554800bbf0 con 0x7f55580fdff0 2024-04-22T21:07:20.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.697+0000 7f554e7fc700 1 -- 172.21.15.38:0/2445963735 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f554801c630 con 0x7f55580fdff0 2024-04-22T21:07:20.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.697+0000 7f555fcaa700 1 -- 172.21.15.38:0/2445963735 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5558113600 con 0x7f55580fdff0 2024-04-22T21:07:20.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.698+0000 7f555fcaa700 1 -- 172.21.15.38:0/2445963735 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f555804c710 con 0x7f55580fdff0 2024-04-22T21:07:20.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.699+0000 7f554e7fc700 1 -- 172.21.15.38:0/2445963735 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f554800bd60 con 0x7f55580fdff0 2024-04-22T21:07:20.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.699+0000 7f554e7fc700 1 --2- 172.21.15.38:0/2445963735 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f55440692a0 0x7f554406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:20.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.699+0000 7f555da46700 1 --2- 172.21.15.38:0/2445963735 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f55440692a0 0x7f554406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:20.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.699+0000 7f554e7fc700 1 -- 172.21.15.38:0/2445963735 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f55480882b0 con 0x7f55580fdff0 2024-04-22T21:07:20.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.699+0000 7f555da46700 1 --2- 172.21.15.38:0/2445963735 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f55440692a0 0x7f554406b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5554006fd0 tx=0x7f5554008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:20.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.703+0000 7f554e7fc700 1 -- 172.21.15.38:0/2445963735 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f5548058460 con 0x7f55580fdff0 2024-04-22T21:07:20.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.906+0000 7f555fcaa700 1 -- 172.21.15.38:0/2445963735 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5558102290 con 0x7f55440692a0 2024-04-22T21:07:20.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.908+0000 7f554e7fc700 1 -- 172.21.15.38:0/2445963735 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f5558102290 con 0x7f55440692a0 2024-04-22T21:07:20.904 INFO:teuthology.orchestra.run.smithi038.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-22T21:07:20.904 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN 3726G No 2m ago Has a FileSystem, LVM detected 2024-04-22T21:07:20.904 INFO:teuthology.orchestra.run.smithi038.stdout:smithi098 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010H400BGN 372G No 68s ago Has a FileSystem, LVM detected 2024-04-22T21:07:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.909+0000 7f555fcaa700 1 -- 172.21.15.38:0/2445963735 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f55440692a0 msgr2=0x7f554406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.909+0000 7f555fcaa700 1 --2- 172.21.15.38:0/2445963735 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f55440692a0 0x7f554406b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5554006fd0 tx=0x7f5554008040 comp rx=0 tx=0).stop 2024-04-22T21:07:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.909+0000 7f555fcaa700 1 -- 172.21.15.38:0/2445963735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55580fdff0 msgr2=0x7f555810de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.910+0000 7f555fcaa700 1 --2- 172.21.15.38:0/2445963735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55580fdff0 0x7f555810de60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5548009150 tx=0x7f5548009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.910+0000 7f555fcaa700 1 -- 172.21.15.38:0/2445963735 shutdown_connections 2024-04-22T21:07:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.910+0000 7f555fcaa700 1 --2- 172.21.15.38:0/2445963735 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f55440692a0 0x7f554406b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.910+0000 7f555fcaa700 1 --2- 172.21.15.38:0/2445963735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55580fce80 0x7f555810d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.910+0000 7f555fcaa700 1 --2- 172.21.15.38:0/2445963735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55580fdff0 0x7f555810de60 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.910+0000 7f555fcaa700 1 -- 172.21.15.38:0/2445963735 >> 172.21.15.38:0/2445963735 conn(0x7f55580f8580 msgr2=0x7f55580f9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.910+0000 7f555fcaa700 1 -- 172.21.15.38:0/2445963735 shutdown_connections 2024-04-22T21:07:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:20.910+0000 7f555fcaa700 1 -- 172.21.15.38:0/2445963735 wait complete. 2024-04-22T21:07:21.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:21 smithi098 ceph-mon[34053]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:21.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:21 smithi038 ceph-mon[26374]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:21.571 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T21:07:21.581 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T21:07:21.581 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-22T21:07:21.848 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:22.239 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:22 smithi038 ceph-mon[26374]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:07:22.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:22 smithi098 ceph-mon[34053]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:07:23.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.214+0000 7f651c605700 1 -- 172.21.15.38:0/1081668683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65140fdff0 msgr2=0x7f65140fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:23.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.214+0000 7f651c605700 1 --2- 172.21.15.38:0/1081668683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65140fdff0 0x7f65140fe460 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6504004940 tx=0x7f6504009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:23.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.214+0000 7f651c605700 1 -- 172.21.15.38:0/1081668683 shutdown_connections 2024-04-22T21:07:23.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.214+0000 7f651c605700 1 --2- 172.21.15.38:0/1081668683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65140fdff0 0x7f65140fe460 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:23.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.214+0000 7f651c605700 1 --2- 172.21.15.38:0/1081668683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65140fce80 0x7f65140fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:23.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.214+0000 7f651c605700 1 -- 172.21.15.38:0/1081668683 >> 172.21.15.38:0/1081668683 conn(0x7f65140f8580 msgr2=0x7f65140fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:23.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.215+0000 7f651c605700 1 -- 172.21.15.38:0/1081668683 shutdown_connections 2024-04-22T21:07:23.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.215+0000 7f651c605700 1 -- 172.21.15.38:0/1081668683 wait complete. 2024-04-22T21:07:23.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.215+0000 7f651c605700 1 Processor -- start 2024-04-22T21:07:23.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.215+0000 7f651c605700 1 -- start start 2024-04-22T21:07:23.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.215+0000 7f651c605700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65140fce80 0x7f651410d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:23.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.215+0000 7f651c605700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65140fdff0 0x7f651410de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:23.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.215+0000 7f651c605700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651410e510 con 0x7f65140fdff0 2024-04-22T21:07:23.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.215+0000 7f651c605700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6514112eb0 con 0x7f65140fce80 2024-04-22T21:07:23.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.215+0000 7f6519ba0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65140fdff0 0x7f651410de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:23.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.215+0000 7f6519ba0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65140fdff0 0x7f651410de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57258/0 (socket says 172.21.15.38:57258) 2024-04-22T21:07:23.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.215+0000 7f6519ba0700 1 -- 172.21.15.38:0/938997973 learned_addr learned my addr 172.21.15.38:0/938997973 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:23.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.215+0000 7f651a3a1700 1 --2- 172.21.15.38:0/938997973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65140fce80 0x7f651410d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:23.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.216+0000 7f6519ba0700 1 -- 172.21.15.38:0/938997973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65140fce80 msgr2=0x7f651410d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:23.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.216+0000 7f6519ba0700 1 --2- 172.21.15.38:0/938997973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65140fce80 0x7f651410d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:23.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.216+0000 7f6519ba0700 1 -- 172.21.15.38:0/938997973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6504005040 con 0x7f65140fdff0 2024-04-22T21:07:23.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.216+0000 7f6519ba0700 1 --2- 172.21.15.38:0/938997973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65140fdff0 0x7f651410de80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f650400b3b0 tx=0x7f6504009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:23.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.216+0000 7f650affd700 1 -- 172.21.15.38:0/938997973 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f650400e040 con 0x7f65140fdff0 2024-04-22T21:07:23.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.216+0000 7f651c605700 1 -- 172.21.15.38:0/938997973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65141130b0 con 0x7f65140fdff0 2024-04-22T21:07:23.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.216+0000 7f650affd700 1 -- 172.21.15.38:0/938997973 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f650400bbf0 con 0x7f65140fdff0 2024-04-22T21:07:23.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.216+0000 7f650affd700 1 -- 172.21.15.38:0/938997973 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f650401c630 con 0x7f65140fdff0 2024-04-22T21:07:23.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.216+0000 7f651c605700 1 -- 172.21.15.38:0/938997973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6514113620 con 0x7f65140fdff0 2024-04-22T21:07:23.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.217+0000 7f651c605700 1 -- 172.21.15.38:0/938997973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651404c710 con 0x7f65140fdff0 2024-04-22T21:07:23.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.218+0000 7f650affd700 1 -- 172.21.15.38:0/938997973 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f650400bd60 con 0x7f65140fdff0 2024-04-22T21:07:23.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.218+0000 7f650affd700 1 --2- 172.21.15.38:0/938997973 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f65000692a0 0x7f650006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:23.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.218+0000 7f650affd700 1 -- 172.21.15.38:0/938997973 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6504087b10 con 0x7f65140fdff0 2024-04-22T21:07:23.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.218+0000 7f651a3a1700 1 --2- 172.21.15.38:0/938997973 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f65000692a0 0x7f650006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:23.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.218+0000 7f651a3a1700 1 --2- 172.21.15.38:0/938997973 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f65000692a0 0x7f650006b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6510009bf0 tx=0x7f65100091a0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:23.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.223+0000 7f650affd700 1 -- 172.21.15.38:0/938997973 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6504057c10 con 0x7f65140fdff0 2024-04-22T21:07:23.249 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:23 smithi098 ceph-mon[34053]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:23.249 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:23 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:07:23.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:23.448+0000 7f651c605700 1 -- 172.21.15.38:0/938997973 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6514102310 con 0x7f65000692a0 2024-04-22T21:07:23.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:23 smithi038 ceph-mon[26374]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:23.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:23 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:07:24.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:24 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-22T21:07:24.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:24 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-22T21:07:25.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:25.249+0000 7f650affd700 1 -- 172.21.15.38:0/938997973 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6514102310 con 0x7f65000692a0 2024-04-22T21:07:25.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:25.252+0000 7f651c605700 1 -- 172.21.15.38:0/938997973 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f65000692a0 msgr2=0x7f650006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:25.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:25.252+0000 7f651c605700 1 --2- 172.21.15.38:0/938997973 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f65000692a0 0x7f650006b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6510009bf0 tx=0x7f65100091a0 comp rx=0 tx=0).stop 2024-04-22T21:07:25.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:25.252+0000 7f651c605700 1 -- 172.21.15.38:0/938997973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65140fdff0 msgr2=0x7f651410de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:25.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:25.252+0000 7f651c605700 1 --2- 172.21.15.38:0/938997973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65140fdff0 0x7f651410de80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f650400b3b0 tx=0x7f6504009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:25.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:25.252+0000 7f651c605700 1 -- 172.21.15.38:0/938997973 shutdown_connections 2024-04-22T21:07:25.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:25.252+0000 7f651c605700 1 --2- 172.21.15.38:0/938997973 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f65000692a0 0x7f650006b750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:25.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:25.252+0000 7f651c605700 1 --2- 172.21.15.38:0/938997973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65140fce80 0x7f651410d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:25.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:25.252+0000 7f651c605700 1 --2- 172.21.15.38:0/938997973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65140fdff0 0x7f651410de80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:25.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:25.252+0000 7f651c605700 1 -- 172.21.15.38:0/938997973 >> 172.21.15.38:0/938997973 conn(0x7f65140f8580 msgr2=0x7f65140f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:25.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:25.252+0000 7f651c605700 1 -- 172.21.15.38:0/938997973 shutdown_connections 2024-04-22T21:07:25.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:25.252+0000 7f651c605700 1 -- 172.21.15.38:0/938997973 wait complete. 2024-04-22T21:07:25.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:25 smithi098 ceph-mon[34053]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:07:25.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:25 smithi098 ceph-mon[34053]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:25.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:25 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-22T21:07:25.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:25 smithi098 ceph-mon[34053]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T21:07:25.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:25 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-22T21:07:25.567 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:25 smithi038 ceph-mon[26374]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:07:25.567 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:25 smithi038 ceph-mon[26374]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:25.567 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:25 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-22T21:07:25.568 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:25 smithi038 ceph-mon[26374]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T21:07:25.568 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:25 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-22T21:07:25.568 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:25 smithi038 ceph-3c5f5352-00eb-11ef-bc93-c7b262605968-mon-smithi038[26370]: 2024-04-22T21:07:25.223+0000 7f9ec1d86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T21:07:25.861 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-22T21:07:26.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-22T21:07:26.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T21:07:26.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-22T21:07:26.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T21:07:26.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T21:07:26.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: osdmap e44: 6 total, 6 up, 6 in 2024-04-22T21:07:26.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-22T21:07:26.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: fsmap cephfs:0 2024-04-22T21:07:26.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: Saving service mds.cephfs spec with placement count:4 2024-04-22T21:07:26.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:26.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:07:26.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:07:26.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:07:26.376 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:26.376 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi038.vmivbd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T21:07:26.376 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi038.vmivbd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T21:07:26.376 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:26 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:07:26.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-22T21:07:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T21:07:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-22T21:07:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T21:07:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T21:07:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: osdmap e44: 6 total, 6 up, 6 in 2024-04-22T21:07:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-22T21:07:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: fsmap cephfs:0 2024-04-22T21:07:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: Saving service mds.cephfs spec with placement count:4 2024-04-22T21:07:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:07:26.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:07:26.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:07:26.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:26.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi038.vmivbd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T21:07:26.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi038.vmivbd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T21:07:26.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:26 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:07:26.872 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:27.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:27 smithi038 ceph-mon[26374]: Deploying daemon mds.cephfs.smithi038.vmivbd on smithi038 2024-04-22T21:07:27.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:27 smithi038 ceph-mon[26374]: pgmap v176: 65 pgs: 16 creating+peering, 43 unknown, 6 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:27.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:27 smithi038 ceph-mon[26374]: osdmap e45: 6 total, 6 up, 6 in 2024-04-22T21:07:27.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:27 smithi098 ceph-mon[34053]: Deploying daemon mds.cephfs.smithi038.vmivbd on smithi038 2024-04-22T21:07:27.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:27 smithi098 ceph-mon[34053]: pgmap v176: 65 pgs: 16 creating+peering, 43 unknown, 6 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:27.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:27 smithi098 ceph-mon[34053]: osdmap e45: 6 total, 6 up, 6 in 2024-04-22T21:07:28.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:28 smithi098 ceph-mon[34053]: osdmap e46: 6 total, 6 up, 6 in 2024-04-22T21:07:28.522 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:28 smithi038 ceph-mon[26374]: osdmap e46: 6 total, 6 up, 6 in 2024-04-22T21:07:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.806+0000 7fc287bf0700 1 -- 172.21.15.38:0/3304243388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2800fbd00 msgr2=0x7fc2800fc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.806+0000 7fc287bf0700 1 --2- 172.21.15.38:0/3304243388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2800fbd00 0x7fc2800fc110 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc27c004940 tx=0x7fc27c009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.806+0000 7fc287bf0700 1 -- 172.21.15.38:0/3304243388 shutdown_connections 2024-04-22T21:07:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.806+0000 7fc287bf0700 1 --2- 172.21.15.38:0/3304243388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2800fcfb0 0x7fc2800fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.806+0000 7fc287bf0700 1 --2- 172.21.15.38:0/3304243388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2800fbd00 0x7fc2800fc110 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.806+0000 7fc287bf0700 1 -- 172.21.15.38:0/3304243388 >> 172.21.15.38:0/3304243388 conn(0x7fc2800f7480 msgr2=0x7fc2800f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.806+0000 7fc287bf0700 1 -- 172.21.15.38:0/3304243388 shutdown_connections 2024-04-22T21:07:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.806+0000 7fc287bf0700 1 -- 172.21.15.38:0/3304243388 wait complete. 2024-04-22T21:07:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.806+0000 7fc287bf0700 1 Processor -- start 2024-04-22T21:07:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.806+0000 7fc287bf0700 1 -- start start 2024-04-22T21:07:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.806+0000 7fc287bf0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2800fbd00 0x7fc280066d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.806+0000 7fc287bf0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2800fcfb0 0x7fc2801127a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.806+0000 7fc287bf0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc280112ce0 con 0x7fc2800fcfb0 2024-04-22T21:07:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.806+0000 7fc287bf0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc280112e20 con 0x7fc2800fbd00 2024-04-22T21:07:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.807+0000 7fc28518b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2800fcfb0 0x7fc2801127a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.807+0000 7fc28518b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2800fcfb0 0x7fc2801127a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57284/0 (socket says 172.21.15.38:57284) 2024-04-22T21:07:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.807+0000 7fc28518b700 1 -- 172.21.15.38:0/476899603 learned_addr learned my addr 172.21.15.38:0/476899603 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.807+0000 7fc28598c700 1 --2- 172.21.15.38:0/476899603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2800fbd00 0x7fc280066d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.807+0000 7fc28518b700 1 -- 172.21.15.38:0/476899603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2800fbd00 msgr2=0x7fc280066d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.807+0000 7fc28518b700 1 --2- 172.21.15.38:0/476899603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2800fbd00 0x7fc280066d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.807+0000 7fc28518b700 1 -- 172.21.15.38:0/476899603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc27c005040 con 0x7fc2800fcfb0 2024-04-22T21:07:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.807+0000 7fc28518b700 1 --2- 172.21.15.38:0/476899603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2800fcfb0 0x7fc2801127a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc270002a50 tx=0x7fc2700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.807+0000 7fc2767fc700 1 -- 172.21.15.38:0/476899603 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc270007cc0 con 0x7fc2800fcfb0 2024-04-22T21:07:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.807+0000 7fc287bf0700 1 -- 172.21.15.38:0/476899603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2801130b0 con 0x7fc2800fcfb0 2024-04-22T21:07:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.807+0000 7fc2767fc700 1 -- 172.21.15.38:0/476899603 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc27000f040 con 0x7fc2800fcfb0 2024-04-22T21:07:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.807+0000 7fc2767fc700 1 -- 172.21.15.38:0/476899603 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc27000d9d0 con 0x7fc2800fcfb0 2024-04-22T21:07:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.807+0000 7fc287bf0700 1 -- 172.21.15.38:0/476899603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc280113670 con 0x7fc2800fcfb0 2024-04-22T21:07:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.808+0000 7fc287bf0700 1 -- 172.21.15.38:0/476899603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28004c710 con 0x7fc2800fcfb0 2024-04-22T21:07:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.810+0000 7fc2767fc700 1 -- 172.21.15.38:0/476899603 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fc27000dc40 con 0x7fc2800fcfb0 2024-04-22T21:07:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.810+0000 7fc2767fc700 1 --2- 172.21.15.38:0/476899603 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc26c0692a0 0x7fc26c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.810+0000 7fc2767fc700 1 -- 172.21.15.38:0/476899603 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fc270087590 con 0x7fc2800fcfb0 2024-04-22T21:07:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.810+0000 7fc28598c700 1 --2- 172.21.15.38:0/476899603 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc26c0692a0 0x7fc26c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.811+0000 7fc28598c700 1 --2- 172.21.15.38:0/476899603 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc26c0692a0 0x7fc26c06b750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc27c00b840 tx=0x7fc27c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:28.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:28.814+0000 7fc2767fc700 1 -- 172.21.15.38:0/476899603 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fc270057350 con 0x7fc2800fcfb0 2024-04-22T21:07:29.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:29.055+0000 7fc287bf0700 1 -- 172.21.15.38:0/476899603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc2801047f0 con 0x7fc2800fcfb0 2024-04-22T21:07:29.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:29.055+0000 7fc2767fc700 1 -- 172.21.15.38:0/476899603 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7fc270056ee0 con 0x7fc2800fcfb0 2024-04-22T21:07:29.052 INFO:teuthology.orchestra.run.smithi038.stdout:e2 2024-04-22T21:07:29.052 INFO:teuthology.orchestra.run.smithi038.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T21:07:29.052 INFO:teuthology.orchestra.run.smithi038.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T21:07:29.052 INFO:teuthology.orchestra.run.smithi038.stdout:legacy client fscid: 1 2024-04-22T21:07:29.052 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:07:29.052 INFO:teuthology.orchestra.run.smithi038.stdout:Filesystem 'cephfs' (1) 2024-04-22T21:07:29.052 INFO:teuthology.orchestra.run.smithi038.stdout:fs_name cephfs 2024-04-22T21:07:29.052 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 2 2024-04-22T21:07:29.052 INFO:teuthology.orchestra.run.smithi038.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-22T21:07:29.052 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-04-22T21:07:25.224291+0000 2024-04-22T21:07:29.052 INFO:teuthology.orchestra.run.smithi038.stdout:modified 2024-04-22T21:07:25.224362+0000 2024-04-22T21:07:29.052 INFO:teuthology.orchestra.run.smithi038.stdout:tableserver 0 2024-04-22T21:07:29.052 INFO:teuthology.orchestra.run.smithi038.stdout:root 0 2024-04-22T21:07:29.053 INFO:teuthology.orchestra.run.smithi038.stdout:session_timeout 60 2024-04-22T21:07:29.053 INFO:teuthology.orchestra.run.smithi038.stdout:session_autoclose 300 2024-04-22T21:07:29.053 INFO:teuthology.orchestra.run.smithi038.stdout:max_file_size 1099511627776 2024-04-22T21:07:29.053 INFO:teuthology.orchestra.run.smithi038.stdout:required_client_features {} 2024-04-22T21:07:29.053 INFO:teuthology.orchestra.run.smithi038.stdout:last_failure 0 2024-04-22T21:07:29.053 INFO:teuthology.orchestra.run.smithi038.stdout:last_failure_osd_epoch 0 2024-04-22T21:07:29.053 INFO:teuthology.orchestra.run.smithi038.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T21:07:29.053 INFO:teuthology.orchestra.run.smithi038.stdout:max_mds 1 2024-04-22T21:07:29.053 INFO:teuthology.orchestra.run.smithi038.stdout:in 2024-04-22T21:07:29.053 INFO:teuthology.orchestra.run.smithi038.stdout:up {} 2024-04-22T21:07:29.053 INFO:teuthology.orchestra.run.smithi038.stdout:failed 2024-04-22T21:07:29.053 INFO:teuthology.orchestra.run.smithi038.stdout:damaged 2024-04-22T21:07:29.053 INFO:teuthology.orchestra.run.smithi038.stdout:stopped 2024-04-22T21:07:29.053 INFO:teuthology.orchestra.run.smithi038.stdout:data_pools [3] 2024-04-22T21:07:29.054 INFO:teuthology.orchestra.run.smithi038.stdout:metadata_pool 2 2024-04-22T21:07:29.054 INFO:teuthology.orchestra.run.smithi038.stdout:inline_data disabled 2024-04-22T21:07:29.054 INFO:teuthology.orchestra.run.smithi038.stdout:balancer 2024-04-22T21:07:29.054 INFO:teuthology.orchestra.run.smithi038.stdout:standby_count_wanted 0 2024-04-22T21:07:29.054 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:07:29.054 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:07:29.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:29.057+0000 7fc287bf0700 1 -- 172.21.15.38:0/476899603 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc26c0692a0 msgr2=0x7fc26c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:29.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:29.057+0000 7fc287bf0700 1 --2- 172.21.15.38:0/476899603 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc26c0692a0 0x7fc26c06b750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc27c00b840 tx=0x7fc27c009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:29.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:29.057+0000 7fc287bf0700 1 -- 172.21.15.38:0/476899603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2800fcfb0 msgr2=0x7fc2801127a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:29.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:29.057+0000 7fc287bf0700 1 --2- 172.21.15.38:0/476899603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2800fcfb0 0x7fc2801127a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc270002a50 tx=0x7fc2700046b0 comp rx=0 tx=0).stop 2024-04-22T21:07:29.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:29.057+0000 7fc287bf0700 1 -- 172.21.15.38:0/476899603 shutdown_connections 2024-04-22T21:07:29.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:29.057+0000 7fc287bf0700 1 --2- 172.21.15.38:0/476899603 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc26c0692a0 0x7fc26c06b750 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:29.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:29.057+0000 7fc287bf0700 1 --2- 172.21.15.38:0/476899603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2800fbd00 0x7fc280066d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:29.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:29.057+0000 7fc287bf0700 1 --2- 172.21.15.38:0/476899603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2800fcfb0 0x7fc2801127a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:29.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:29.057+0000 7fc287bf0700 1 -- 172.21.15.38:0/476899603 >> 172.21.15.38:0/476899603 conn(0x7fc2800f7480 msgr2=0x7fc2800f8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:29.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:29.057+0000 7fc287bf0700 1 -- 172.21.15.38:0/476899603 shutdown_connections 2024-04-22T21:07:29.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:29.057+0000 7fc287bf0700 1 -- 172.21.15.38:0/476899603 wait complete. 2024-04-22T21:07:29.056 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 2 2024-04-22T21:07:29.349 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:29 smithi098 ceph-mon[34053]: pgmap v179: 65 pgs: 11 creating+activating, 16 creating+peering, 9 unknown, 29 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:29.349 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:29 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:29.350 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:29 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:29.350 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:29 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:29.350 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:29 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi098.mipeta", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T21:07:29.350 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:29 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi098.mipeta", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T21:07:29.350 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:29 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:07:29.350 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:29 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/476899603' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T21:07:29.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:29 smithi038 ceph-mon[26374]: pgmap v179: 65 pgs: 11 creating+activating, 16 creating+peering, 9 unknown, 29 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:07:29.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:29 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:29.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:29 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:29.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:29 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:29.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:29 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi098.mipeta", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T21:07:29.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:29 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi098.mipeta", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T21:07:29.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:29 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:07:29.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:29 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/476899603' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T21:07:29.782 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T21:07:29.792 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T21:07:29.793 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-04-22T21:07:30.065 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:30.438 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:30 smithi098 ceph-mon[34053]: Deploying daemon mds.cephfs.smithi098.mipeta on smithi098 2024-04-22T21:07:30.438 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:30 smithi098 ceph-mon[34053]: mds.? [v2:172.21.15.38:6826/3707300297,v1:172.21.15.38:6827/3707300297] up:boot 2024-04-22T21:07:30.438 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:30 smithi098 ceph-mon[34053]: daemon mds.cephfs.smithi038.vmivbd assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-22T21:07:30.438 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:30 smithi098 ceph-mon[34053]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T21:07:30.438 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:30 smithi098 ceph-mon[34053]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T21:07:30.438 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:30 smithi098 ceph-mon[34053]: Cluster is now healthy 2024-04-22T21:07:30.439 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:30 smithi098 ceph-mon[34053]: fsmap cephfs:0 1 up:standby 2024-04-22T21:07:30.439 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:30 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi038.vmivbd"}]: dispatch 2024-04-22T21:07:30.439 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:30 smithi098 ceph-mon[34053]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:creating} 2024-04-22T21:07:30.439 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:30 smithi098 ceph-mon[34053]: daemon mds.cephfs.smithi038.vmivbd is now active in filesystem cephfs as rank 0 2024-04-22T21:07:30.506 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:30 smithi038 ceph-mon[26374]: Deploying daemon mds.cephfs.smithi098.mipeta on smithi098 2024-04-22T21:07:30.506 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:30 smithi038 ceph-mon[26374]: mds.? [v2:172.21.15.38:6826/3707300297,v1:172.21.15.38:6827/3707300297] up:boot 2024-04-22T21:07:30.506 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:30 smithi038 ceph-mon[26374]: daemon mds.cephfs.smithi038.vmivbd assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-22T21:07:30.506 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:30 smithi038 ceph-mon[26374]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T21:07:30.506 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:30 smithi038 ceph-mon[26374]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T21:07:30.507 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:30 smithi038 ceph-mon[26374]: Cluster is now healthy 2024-04-22T21:07:30.507 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:30 smithi038 ceph-mon[26374]: fsmap cephfs:0 1 up:standby 2024-04-22T21:07:30.507 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:30 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi038.vmivbd"}]: dispatch 2024-04-22T21:07:30.507 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:30 smithi038 ceph-mon[26374]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:creating} 2024-04-22T21:07:30.507 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:30 smithi038 ceph-mon[26374]: daemon mds.cephfs.smithi038.vmivbd is now active in filesystem cephfs as rank 0 2024-04-22T21:07:31.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:31 smithi038 ceph-mon[26374]: pgmap v180: 65 pgs: 11 creating+activating, 4 creating+peering, 50 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 840 B/s wr, 3 op/s 2024-04-22T21:07:31.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:31 smithi038 ceph-mon[26374]: mds.? [v2:172.21.15.38:6826/3707300297,v1:172.21.15.38:6827/3707300297] up:active 2024-04-22T21:07:31.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:31 smithi038 ceph-mon[26374]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:active} 2024-04-22T21:07:31.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:31 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:31.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:31 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:31.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:31 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:31.460 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:31 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi038.hfegws", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T21:07:31.460 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:31 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi038.hfegws", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T21:07:31.460 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:31 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:07:31.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.686+0000 7fbe2ac65700 1 -- 172.21.15.38:0/1391484516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe240fbce0 msgr2=0x7fbe240fc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:31.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.686+0000 7fbe2ac65700 1 --2- 172.21.15.38:0/1391484516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe240fbce0 0x7fbe240fc0f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbe18004940 tx=0x7fbe18009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:31.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.686+0000 7fbe2ac65700 1 -- 172.21.15.38:0/1391484516 shutdown_connections 2024-04-22T21:07:31.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.686+0000 7fbe2ac65700 1 --2- 172.21.15.38:0/1391484516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe240fcf90 0x7fbe240fd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:31.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.686+0000 7fbe2ac65700 1 --2- 172.21.15.38:0/1391484516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe240fbce0 0x7fbe240fc0f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:31.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.686+0000 7fbe2ac65700 1 -- 172.21.15.38:0/1391484516 >> 172.21.15.38:0/1391484516 conn(0x7fbe240f7440 msgr2=0x7fbe240f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:31.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.686+0000 7fbe2ac65700 1 -- 172.21.15.38:0/1391484516 shutdown_connections 2024-04-22T21:07:31.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.686+0000 7fbe2ac65700 1 -- 172.21.15.38:0/1391484516 wait complete. 2024-04-22T21:07:31.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.687+0000 7fbe2ac65700 1 Processor -- start 2024-04-22T21:07:31.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.687+0000 7fbe2ac65700 1 -- start start 2024-04-22T21:07:31.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.687+0000 7fbe2ac65700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe240fbce0 0x7fbe2410c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:31.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.687+0000 7fbe2ac65700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe240fcf90 0x7fbe2410cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:31.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.687+0000 7fbe2ac65700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2410d3e0 con 0x7fbe240fcf90 2024-04-22T21:07:31.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.687+0000 7fbe2ac65700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2410dc30 con 0x7fbe240fbce0 2024-04-22T21:07:31.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.687+0000 7fbe23fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe240fcf90 0x7fbe2410cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:31.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.687+0000 7fbe23fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe240fcf90 0x7fbe2410cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57304/0 (socket says 172.21.15.38:57304) 2024-04-22T21:07:31.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.687+0000 7fbe23fff700 1 -- 172.21.15.38:0/3666828413 learned_addr learned my addr 172.21.15.38:0/3666828413 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:31.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.688+0000 7fbe28a01700 1 --2- 172.21.15.38:0/3666828413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe240fbce0 0x7fbe2410c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:31.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.688+0000 7fbe23fff700 1 -- 172.21.15.38:0/3666828413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe240fbce0 msgr2=0x7fbe2410c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:31.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.688+0000 7fbe23fff700 1 --2- 172.21.15.38:0/3666828413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe240fbce0 0x7fbe2410c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:31.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.688+0000 7fbe23fff700 1 -- 172.21.15.38:0/3666828413 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe18005040 con 0x7fbe240fcf90 2024-04-22T21:07:31.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.688+0000 7fbe23fff700 1 --2- 172.21.15.38:0/3666828413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe240fcf90 0x7fbe2410cde0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbe1000eb40 tx=0x7fbe1000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:31.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.688+0000 7fbe217fa700 1 -- 172.21.15.38:0/3666828413 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe10007cc0 con 0x7fbe240fcf90 2024-04-22T21:07:31.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.688+0000 7fbe217fa700 1 -- 172.21.15.38:0/3666828413 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe1000f040 con 0x7fbe240fcf90 2024-04-22T21:07:31.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.688+0000 7fbe217fa700 1 -- 172.21.15.38:0/3666828413 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe1000ca10 con 0x7fbe240fcf90 2024-04-22T21:07:31.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.688+0000 7fbe2ac65700 1 -- 172.21.15.38:0/3666828413 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe2410deb0 con 0x7fbe240fcf90 2024-04-22T21:07:31.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.689+0000 7fbe2ac65700 1 -- 172.21.15.38:0/3666828413 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe2410e3a0 con 0x7fbe240fcf90 2024-04-22T21:07:31.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.690+0000 7fbe2ac65700 1 -- 172.21.15.38:0/3666828413 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe2405f5f0 con 0x7fbe240fcf90 2024-04-22T21:07:31.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.690+0000 7fbe217fa700 1 -- 172.21.15.38:0/3666828413 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fbe1000cc80 con 0x7fbe240fcf90 2024-04-22T21:07:31.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.690+0000 7fbe217fa700 1 --2- 172.21.15.38:0/3666828413 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fbe1406b4b0 0x7fbe1406d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:31.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.691+0000 7fbe217fa700 1 -- 172.21.15.38:0/3666828413 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fbe10086df0 con 0x7fbe240fcf90 2024-04-22T21:07:31.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.691+0000 7fbe28a01700 1 --2- 172.21.15.38:0/3666828413 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fbe1406b4b0 0x7fbe1406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:31.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.691+0000 7fbe28a01700 1 --2- 172.21.15.38:0/3666828413 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fbe1406b4b0 0x7fbe1406d960 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009b30 tx=0x7fbe18009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:31.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.695+0000 7fbe217fa700 1 -- 172.21.15.38:0/3666828413 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fbe1004e250 con 0x7fbe240fcf90 2024-04-22T21:07:31.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:31 smithi098 ceph-mon[34053]: pgmap v180: 65 pgs: 11 creating+activating, 4 creating+peering, 50 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 840 B/s wr, 3 op/s 2024-04-22T21:07:31.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:31 smithi098 ceph-mon[34053]: mds.? [v2:172.21.15.38:6826/3707300297,v1:172.21.15.38:6827/3707300297] up:active 2024-04-22T21:07:31.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:31 smithi098 ceph-mon[34053]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:active} 2024-04-22T21:07:31.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:31 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:31.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:31 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:31.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:31 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:31.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:31 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi038.hfegws", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T21:07:31.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:31 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi038.hfegws", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T21:07:31.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:31 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:07:31.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:31.944+0000 7fbe2ac65700 1 -- 172.21.15.38:0/3666828413 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fbe24104800 con 0x7fbe240fcf90 2024-04-22T21:07:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:32.275+0000 7fbe217fa700 1 -- 172.21.15.38:0/3666828413 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7fbe10056b80 con 0x7fbe240fcf90 2024-04-22T21:07:32.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:32.277+0000 7fbe2ac65700 1 -- 172.21.15.38:0/3666828413 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fbe1406b4b0 msgr2=0x7fbe1406d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:32.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:32.277+0000 7fbe2ac65700 1 --2- 172.21.15.38:0/3666828413 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fbe1406b4b0 0x7fbe1406d960 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009b30 tx=0x7fbe18009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:32.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:32.277+0000 7fbe2ac65700 1 -- 172.21.15.38:0/3666828413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe240fcf90 msgr2=0x7fbe2410cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:32.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:32.277+0000 7fbe2ac65700 1 --2- 172.21.15.38:0/3666828413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe240fcf90 0x7fbe2410cde0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbe1000eb40 tx=0x7fbe1000b040 comp rx=0 tx=0).stop 2024-04-22T21:07:32.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:32.278+0000 7fbe2ac65700 1 -- 172.21.15.38:0/3666828413 shutdown_connections 2024-04-22T21:07:32.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:32.278+0000 7fbe2ac65700 1 --2- 172.21.15.38:0/3666828413 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fbe1406b4b0 0x7fbe1406d960 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:32.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:32.278+0000 7fbe2ac65700 1 --2- 172.21.15.38:0/3666828413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe240fbce0 0x7fbe2410c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:32.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:32.278+0000 7fbe2ac65700 1 --2- 172.21.15.38:0/3666828413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe240fcf90 0x7fbe2410cde0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:32.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:32.278+0000 7fbe2ac65700 1 -- 172.21.15.38:0/3666828413 >> 172.21.15.38:0/3666828413 conn(0x7fbe240f7440 msgr2=0x7fbe240f8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:32.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:32.278+0000 7fbe2ac65700 1 -- 172.21.15.38:0/3666828413 shutdown_connections 2024-04-22T21:07:32.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:32.278+0000 7fbe2ac65700 1 -- 172.21.15.38:0/3666828413 wait complete. 2024-04-22T21:07:32.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:32 smithi038 ceph-mon[26374]: Deploying daemon mds.cephfs.smithi038.hfegws on smithi038 2024-04-22T21:07:32.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:32 smithi038 ceph-mon[26374]: mds.? [v2:172.21.15.98:6824/2824268703,v1:172.21.15.98:6825/2824268703] up:boot 2024-04-22T21:07:32.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:32 smithi038 ceph-mon[26374]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:active} 1 up:standby 2024-04-22T21:07:32.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:32 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi098.mipeta"}]: dispatch 2024-04-22T21:07:32.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:32 smithi038 ceph-mon[26374]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:active} 1 up:standby 2024-04-22T21:07:32.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:32 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3666828413' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-04-22T21:07:32.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:32 smithi098 ceph-mon[34053]: Deploying daemon mds.cephfs.smithi038.hfegws on smithi038 2024-04-22T21:07:32.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:32 smithi098 ceph-mon[34053]: mds.? [v2:172.21.15.98:6824/2824268703,v1:172.21.15.98:6825/2824268703] up:boot 2024-04-22T21:07:32.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:32 smithi098 ceph-mon[34053]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:active} 1 up:standby 2024-04-22T21:07:32.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:32 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi098.mipeta"}]: dispatch 2024-04-22T21:07:32.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:32 smithi098 ceph-mon[34053]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:active} 1 up:standby 2024-04-22T21:07:32.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:32 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/3666828413' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-04-22T21:07:32.831 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T21:07:32.842 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T21:07:32.842 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-04-22T21:07:33.305 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:33.429 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:33 smithi038 ceph-mon[26374]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-04-22T21:07:33.429 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:33 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3666828413' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-04-22T21:07:33.429 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:33 smithi038 ceph-mon[26374]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:active} 1 up:standby 2024-04-22T21:07:33.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:33 smithi098 ceph-mon[34053]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-04-22T21:07:33.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:33 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/3666828413' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-04-22T21:07:33.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:33 smithi098 ceph-mon[34053]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:active} 1 up:standby 2024-04-22T21:07:34.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.911+0000 7feb775d5700 1 -- 172.21.15.38:0/1673528126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb700fdff0 msgr2=0x7feb700fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:34.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.911+0000 7feb775d5700 1 --2- 172.21.15.38:0/1673528126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb700fdff0 0x7feb700fe460 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feb60004940 tx=0x7feb60009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:34.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.912+0000 7feb775d5700 1 -- 172.21.15.38:0/1673528126 shutdown_connections 2024-04-22T21:07:34.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.912+0000 7feb775d5700 1 --2- 172.21.15.38:0/1673528126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb700fdff0 0x7feb700fe460 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:34.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.912+0000 7feb775d5700 1 --2- 172.21.15.38:0/1673528126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb700fce80 0x7feb700fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:34.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.912+0000 7feb775d5700 1 -- 172.21.15.38:0/1673528126 >> 172.21.15.38:0/1673528126 conn(0x7feb700f8580 msgr2=0x7feb700fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:34.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.912+0000 7feb775d5700 1 -- 172.21.15.38:0/1673528126 shutdown_connections 2024-04-22T21:07:34.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.912+0000 7feb775d5700 1 -- 172.21.15.38:0/1673528126 wait complete. 2024-04-22T21:07:34.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.912+0000 7feb775d5700 1 Processor -- start 2024-04-22T21:07:34.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.912+0000 7feb775d5700 1 -- start start 2024-04-22T21:07:34.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.912+0000 7feb775d5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb700fce80 0x7feb7010d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:34.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.912+0000 7feb775d5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb700fdff0 0x7feb7010de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:34.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.912+0000 7feb775d5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7010e470 con 0x7feb700fce80 2024-04-22T21:07:34.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.912+0000 7feb775d5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7010e5e0 con 0x7feb700fdff0 2024-04-22T21:07:34.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.913+0000 7feb75371700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb700fce80 0x7feb7010d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:34.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.913+0000 7feb75371700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb700fce80 0x7feb7010d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55990/0 (socket says 172.21.15.38:55990) 2024-04-22T21:07:34.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.913+0000 7feb75371700 1 -- 172.21.15.38:0/1201394502 learned_addr learned my addr 172.21.15.38:0/1201394502 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:34.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.913+0000 7feb74b70700 1 --2- 172.21.15.38:0/1201394502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb700fdff0 0x7feb7010de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:34.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.913+0000 7feb75371700 1 -- 172.21.15.38:0/1201394502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb700fdff0 msgr2=0x7feb7010de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:34.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.913+0000 7feb75371700 1 --2- 172.21.15.38:0/1201394502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb700fdff0 0x7feb7010de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:34.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.913+0000 7feb75371700 1 -- 172.21.15.38:0/1201394502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb60005040 con 0x7feb700fce80 2024-04-22T21:07:34.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.913+0000 7feb75371700 1 --2- 172.21.15.38:0/1201394502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb700fce80 0x7feb7010d930 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7feb6c002a50 tx=0x7feb6c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:34.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.913+0000 7feb65ffb700 1 -- 172.21.15.38:0/1201394502 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb6c007cc0 con 0x7feb700fce80 2024-04-22T21:07:34.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.913+0000 7feb65ffb700 1 -- 172.21.15.38:0/1201394502 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb6c00f040 con 0x7feb700fce80 2024-04-22T21:07:34.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.913+0000 7feb775d5700 1 -- 172.21.15.38:0/1201394502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb70112fe0 con 0x7feb700fce80 2024-04-22T21:07:34.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.913+0000 7feb65ffb700 1 -- 172.21.15.38:0/1201394502 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb6c00da30 con 0x7feb700fce80 2024-04-22T21:07:34.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.913+0000 7feb775d5700 1 -- 172.21.15.38:0/1201394502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb70113570 con 0x7feb700fce80 2024-04-22T21:07:34.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.914+0000 7feb775d5700 1 -- 172.21.15.38:0/1201394502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7004c710 con 0x7feb700fce80 2024-04-22T21:07:34.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.915+0000 7feb65ffb700 1 -- 172.21.15.38:0/1201394502 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7feb6c00db90 con 0x7feb700fce80 2024-04-22T21:07:34.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.915+0000 7feb65ffb700 1 --2- 172.21.15.38:0/1201394502 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7feb5c0692a0 0x7feb5c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:34.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.915+0000 7feb65ffb700 1 -- 172.21.15.38:0/1201394502 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7feb6c086e60 con 0x7feb700fce80 2024-04-22T21:07:34.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.915+0000 7feb74b70700 1 --2- 172.21.15.38:0/1201394502 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7feb5c0692a0 0x7feb5c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:34.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.916+0000 7feb74b70700 1 --2- 172.21.15.38:0/1201394502 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7feb5c0692a0 0x7feb5c06b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7feb6000b970 tx=0x7feb60009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:34.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:34.920+0000 7feb65ffb700 1 -- 172.21.15.38:0/1201394502 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7feb6c056c20 con 0x7feb700fce80 2024-04-22T21:07:35.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:35.188+0000 7feb775d5700 1 -- 172.21.15.38:0/1201394502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7feb701058e0 con 0x7feb700fce80 2024-04-22T21:07:35.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:35.294+0000 7feb65ffb700 1 -- 172.21.15.38:0/1201394502 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7feb6c0567b0 con 0x7feb700fce80 2024-04-22T21:07:35.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:35.296+0000 7feb775d5700 1 -- 172.21.15.38:0/1201394502 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7feb5c0692a0 msgr2=0x7feb5c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:35.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:35.296+0000 7feb775d5700 1 --2- 172.21.15.38:0/1201394502 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7feb5c0692a0 0x7feb5c06b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7feb6000b970 tx=0x7feb60009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:35.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:35.296+0000 7feb775d5700 1 -- 172.21.15.38:0/1201394502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb700fce80 msgr2=0x7feb7010d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:35.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:35.296+0000 7feb775d5700 1 --2- 172.21.15.38:0/1201394502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb700fce80 0x7feb7010d930 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7feb6c002a50 tx=0x7feb6c0046b0 comp rx=0 tx=0).stop 2024-04-22T21:07:35.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:35.296+0000 7feb775d5700 1 -- 172.21.15.38:0/1201394502 shutdown_connections 2024-04-22T21:07:35.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:35.296+0000 7feb775d5700 1 --2- 172.21.15.38:0/1201394502 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7feb5c0692a0 0x7feb5c06b750 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:35.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:35.296+0000 7feb775d5700 1 --2- 172.21.15.38:0/1201394502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb700fce80 0x7feb7010d930 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:35.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:35.296+0000 7feb775d5700 1 --2- 172.21.15.38:0/1201394502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb700fdff0 0x7feb7010de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:35.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:35.296+0000 7feb775d5700 1 -- 172.21.15.38:0/1201394502 >> 172.21.15.38:0/1201394502 conn(0x7feb700f8580 msgr2=0x7feb700f9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:35.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:35.296+0000 7feb775d5700 1 -- 172.21.15.38:0/1201394502 shutdown_connections 2024-04-22T21:07:35.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:35.296+0000 7feb775d5700 1 -- 172.21.15.38:0/1201394502 wait complete. 2024-04-22T21:07:35.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:35 smithi038 ceph-mon[26374]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-04-22T21:07:35.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:35 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1201394502' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-04-22T21:07:35.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:35 smithi098 ceph-mon[34053]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-04-22T21:07:35.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:35 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/1201394502' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-04-22T21:07:35.970 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T21:07:35.981 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T21:07:35.981 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-04-22T21:07:36.248 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:36.549 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:36 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1201394502' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-04-22T21:07:36.549 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:36 smithi038 ceph-mon[26374]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:active} 1 up:standby 2024-04-22T21:07:36.549 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:36 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:36.549 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:36 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:36.549 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:36 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:36.549 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:36 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi098.izcoza", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T21:07:36.549 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:36 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi098.izcoza", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T21:07:36.550 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:36 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:07:36.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:36 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/1201394502' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-04-22T21:07:36.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:36 smithi098 ceph-mon[34053]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:active} 1 up:standby 2024-04-22T21:07:36.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:36 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:36.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:36 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:36.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:36 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:36.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:36 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi098.izcoza", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T21:07:36.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:36 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi098.izcoza", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T21:07:36.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:36 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:07:37.529 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:37 smithi038 ceph-mon[26374]: Deploying daemon mds.cephfs.smithi098.izcoza on smithi098 2024-04-22T21:07:37.529 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:37 smithi038 ceph-mon[26374]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-22T21:07:37.529 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:37 smithi038 ceph-mon[26374]: mds.? [v2:172.21.15.38:6828/92551647,v1:172.21.15.38:6829/92551647] up:boot 2024-04-22T21:07:37.529 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:37 smithi038 ceph-mon[26374]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:active} 2 up:standby 2024-04-22T21:07:37.529 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:37 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi038.hfegws"}]: dispatch 2024-04-22T21:07:37.592 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:37 smithi098 ceph-mon[34053]: Deploying daemon mds.cephfs.smithi098.izcoza on smithi098 2024-04-22T21:07:37.593 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:37 smithi098 ceph-mon[34053]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-22T21:07:37.593 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:37 smithi098 ceph-mon[34053]: mds.? [v2:172.21.15.38:6828/92551647,v1:172.21.15.38:6829/92551647] up:boot 2024-04-22T21:07:37.593 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:37 smithi098 ceph-mon[34053]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:active} 2 up:standby 2024-04-22T21:07:37.593 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:37 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi038.hfegws"}]: dispatch 2024-04-22T21:07:37.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.736+0000 7fbefa068700 1 -- 172.21.15.38:0/1951920983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbef40fbd40 msgr2=0x7fbef40fc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:37.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.736+0000 7fbefa068700 1 --2- 172.21.15.38:0/1951920983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbef40fbd40 0x7fbef40fc150 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbee4004940 tx=0x7fbee4009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:37.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.736+0000 7fbefa068700 1 -- 172.21.15.38:0/1951920983 shutdown_connections 2024-04-22T21:07:37.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.736+0000 7fbefa068700 1 --2- 172.21.15.38:0/1951920983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbef40fcff0 0x7fbef40fd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:37.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.736+0000 7fbefa068700 1 --2- 172.21.15.38:0/1951920983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbef40fbd40 0x7fbef40fc150 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:37.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.736+0000 7fbefa068700 1 -- 172.21.15.38:0/1951920983 >> 172.21.15.38:0/1951920983 conn(0x7fbef40f74c0 msgr2=0x7fbef40f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:37.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.736+0000 7fbefa068700 1 -- 172.21.15.38:0/1951920983 shutdown_connections 2024-04-22T21:07:37.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.736+0000 7fbefa068700 1 -- 172.21.15.38:0/1951920983 wait complete. 2024-04-22T21:07:37.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.737+0000 7fbefa068700 1 Processor -- start 2024-04-22T21:07:37.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.737+0000 7fbefa068700 1 -- start start 2024-04-22T21:07:37.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.737+0000 7fbefa068700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbef40fbd40 0x7fbef410c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:37.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.737+0000 7fbefa068700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbef40fcff0 0x7fbef410ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:37.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.737+0000 7fbefa068700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef410d400 con 0x7fbef40fbd40 2024-04-22T21:07:37.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.737+0000 7fbefa068700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef410dc50 con 0x7fbef40fcff0 2024-04-22T21:07:37.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.737+0000 7fbef37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbef40fbd40 0x7fbef410c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:37.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.737+0000 7fbef37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbef40fbd40 0x7fbef410c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56026/0 (socket says 172.21.15.38:56026) 2024-04-22T21:07:37.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.737+0000 7fbef37fe700 1 -- 172.21.15.38:0/1986167138 learned_addr learned my addr 172.21.15.38:0/1986167138 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:37.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.738+0000 7fbef37fe700 1 -- 172.21.15.38:0/1986167138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbef40fcff0 msgr2=0x7fbef410ce00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T21:07:37.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.738+0000 7fbef37fe700 1 --2- 172.21.15.38:0/1986167138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbef40fcff0 0x7fbef410ce00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:37.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.738+0000 7fbef37fe700 1 -- 172.21.15.38:0/1986167138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee4005040 con 0x7fbef40fbd40 2024-04-22T21:07:37.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.738+0000 7fbef37fe700 1 --2- 172.21.15.38:0/1986167138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbef40fbd40 0x7fbef410c8c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbee4013870 tx=0x7fbee4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:37.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.738+0000 7fbedbfff700 1 -- 172.21.15.38:0/1986167138 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbee4021070 con 0x7fbef40fbd40 2024-04-22T21:07:37.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.738+0000 7fbedbfff700 1 -- 172.21.15.38:0/1986167138 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbee401c400 con 0x7fbef40fbd40 2024-04-22T21:07:37.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.738+0000 7fbefa068700 1 -- 172.21.15.38:0/1986167138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef410ded0 con 0x7fbef40fbd40 2024-04-22T21:07:37.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.738+0000 7fbedbfff700 1 -- 172.21.15.38:0/1986167138 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbee400e070 con 0x7fbef40fbd40 2024-04-22T21:07:37.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.738+0000 7fbefa068700 1 -- 172.21.15.38:0/1986167138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef410e410 con 0x7fbef40fbd40 2024-04-22T21:07:37.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.739+0000 7fbefa068700 1 -- 172.21.15.38:0/1986167138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef405f5f0 con 0x7fbef40fbd40 2024-04-22T21:07:37.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.739+0000 7fbedbfff700 1 -- 172.21.15.38:0/1986167138 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fbee401ca20 con 0x7fbef40fbd40 2024-04-22T21:07:37.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.740+0000 7fbedbfff700 1 --2- 172.21.15.38:0/1986167138 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fbedc0692a0 0x7fbedc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:37.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.740+0000 7fbedbfff700 1 -- 172.21.15.38:0/1986167138 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fbee4088580 con 0x7fbef40fbd40 2024-04-22T21:07:37.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.740+0000 7fbef2ffd700 1 --2- 172.21.15.38:0/1986167138 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fbedc0692a0 0x7fbedc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:37.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.740+0000 7fbef2ffd700 1 --2- 172.21.15.38:0/1986167138 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fbedc0692a0 0x7fbedc06b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009fd0 tx=0x7fbee800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:37.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.744+0000 7fbedbfff700 1 -- 172.21.15.38:0/1986167138 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fbee4058340 con 0x7fbef40fbd40 2024-04-22T21:07:37.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:37.986+0000 7fbefa068700 1 -- 172.21.15.38:0/1986167138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbef404bed0 con 0x7fbef40fbd40 2024-04-22T21:07:38.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:38.308+0000 7fbedbfff700 1 -- 172.21.15.38:0/1986167138 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v11) v1 ==== 168+0+0 (secure 0 0 0) 0x7fbee4004490 con 0x7fbef40fbd40 2024-04-22T21:07:38.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:38.310+0000 7fbefa068700 1 -- 172.21.15.38:0/1986167138 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fbedc0692a0 msgr2=0x7fbedc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:38.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:38.310+0000 7fbefa068700 1 --2- 172.21.15.38:0/1986167138 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fbedc0692a0 0x7fbedc06b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009fd0 tx=0x7fbee800b040 comp rx=0 tx=0).stop 2024-04-22T21:07:38.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:38.310+0000 7fbefa068700 1 -- 172.21.15.38:0/1986167138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbef40fbd40 msgr2=0x7fbef410c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:38.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:38.310+0000 7fbefa068700 1 --2- 172.21.15.38:0/1986167138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbef40fbd40 0x7fbef410c8c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbee4013870 tx=0x7fbee4009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:38.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:38.310+0000 7fbefa068700 1 -- 172.21.15.38:0/1986167138 shutdown_connections 2024-04-22T21:07:38.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:38.310+0000 7fbefa068700 1 --2- 172.21.15.38:0/1986167138 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fbedc0692a0 0x7fbedc06b750 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:38.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:38.310+0000 7fbefa068700 1 --2- 172.21.15.38:0/1986167138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbef40fbd40 0x7fbef410c8c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:38.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:38.310+0000 7fbefa068700 1 --2- 172.21.15.38:0/1986167138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbef40fcff0 0x7fbef410ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:38.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:38.310+0000 7fbefa068700 1 -- 172.21.15.38:0/1986167138 >> 172.21.15.38:0/1986167138 conn(0x7fbef40f74c0 msgr2=0x7fbef40f8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:38.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:38.310+0000 7fbefa068700 1 -- 172.21.15.38:0/1986167138 shutdown_connections 2024-04-22T21:07:38.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:38.310+0000 7fbefa068700 1 -- 172.21.15.38:0/1986167138 wait complete. 2024-04-22T21:07:38.307 INFO:teuthology.orchestra.run.smithi038.stderr:inline data enabled 2024-04-22T21:07:38.920 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T21:07:38.930 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T21:07:38.930 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-22T21:07:39.016 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:38 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:39.016 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:38 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:39.016 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:38 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/1986167138' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T21:07:39.017 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:38 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:39.017 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:38 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:39.017 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:38 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:39.017 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:38 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:07:39.017 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:38 smithi098 ceph-mon[34053]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-22T21:07:39.017 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:38 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:39.017 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:38 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:07:39.017 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:38 smithi098 ceph-mon[34053]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-22T21:07:39.017 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:38 smithi098 ceph-mon[34053]: mds.? [v2:172.21.15.98:6826/1024898784,v1:172.21.15.98:6827/1024898784] up:boot 2024-04-22T21:07:39.017 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:38 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/1986167138' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T21:07:39.017 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:38 smithi098 ceph-mon[34053]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:active} 3 up:standby 2024-04-22T21:07:39.017 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:38 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi098.izcoza"}]: dispatch 2024-04-22T21:07:39.279 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:39.280 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:39.280 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:38 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1986167138' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T21:07:39.280 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:39.280 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:39.280 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:39.280 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:07:39.280 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:38 smithi038 ceph-mon[26374]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-22T21:07:39.280 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:39.280 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:07:39.280 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:38 smithi038 ceph-mon[26374]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-22T21:07:39.281 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:38 smithi038 ceph-mon[26374]: mds.? [v2:172.21.15.98:6826/1024898784,v1:172.21.15.98:6827/1024898784] up:boot 2024-04-22T21:07:39.281 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:38 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1986167138' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T21:07:39.281 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:38 smithi038 ceph-mon[26374]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:active} 3 up:standby 2024-04-22T21:07:39.281 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi098.izcoza"}]: dispatch 2024-04-22T21:07:39.692 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:40.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:40 smithi038 ceph-mon[26374]: mds.? [v2:172.21.15.98:6824/2824268703,v1:172.21.15.98:6825/2824268703] up:standby 2024-04-22T21:07:40.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:40 smithi038 ceph-mon[26374]: Dropping low affinity active daemon mds.cephfs.smithi038.vmivbd in favor of higher affinity standby. 2024-04-22T21:07:40.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:40 smithi038 ceph-mon[26374]: Replacing daemon mds.cephfs.smithi038.vmivbd as rank 0 with standby daemon mds.cephfs.smithi098.mipeta 2024-04-22T21:07:40.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:40 smithi038 ceph-mon[26374]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T21:07:40.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:40 smithi038 ceph-mon[26374]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:active} 3 up:standby 2024-04-22T21:07:40.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:40 smithi038 ceph-mon[26374]: osdmap e47: 6 total, 6 up, 6 in 2024-04-22T21:07:40.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:40 smithi038 ceph-mon[26374]: fsmap cephfs:1/1 {0=cephfs.smithi098.mipeta=up:replay} 2 up:standby 2024-04-22T21:07:40.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:40 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:40.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:40 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:40.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:40 smithi098 ceph-mon[34053]: mds.? [v2:172.21.15.98:6824/2824268703,v1:172.21.15.98:6825/2824268703] up:standby 2024-04-22T21:07:40.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:40 smithi098 ceph-mon[34053]: Dropping low affinity active daemon mds.cephfs.smithi038.vmivbd in favor of higher affinity standby. 2024-04-22T21:07:40.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:40 smithi098 ceph-mon[34053]: Replacing daemon mds.cephfs.smithi038.vmivbd as rank 0 with standby daemon mds.cephfs.smithi098.mipeta 2024-04-22T21:07:40.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:40 smithi098 ceph-mon[34053]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T21:07:40.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:40 smithi098 ceph-mon[34053]: fsmap cephfs:1 {0=cephfs.smithi038.vmivbd=up:active} 3 up:standby 2024-04-22T21:07:40.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:40 smithi098 ceph-mon[34053]: osdmap e47: 6 total, 6 up, 6 in 2024-04-22T21:07:40.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:40 smithi098 ceph-mon[34053]: fsmap cephfs:1/1 {0=cephfs.smithi098.mipeta=up:replay} 2 up:standby 2024-04-22T21:07:40.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:40 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:40.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:40 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:41.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:41 smithi038 ceph-mon[26374]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 614 B/s rd, 921 B/s wr, 3 op/s 2024-04-22T21:07:41.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:41 smithi038 ceph-mon[26374]: mds.? [v2:172.21.15.98:6824/2824268703,v1:172.21.15.98:6825/2824268703] up:reconnect 2024-04-22T21:07:41.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:41 smithi038 ceph-mon[26374]: mds.? [v2:172.21.15.38:6826/3829229281,v1:172.21.15.38:6827/3829229281] up:boot 2024-04-22T21:07:41.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:41 smithi038 ceph-mon[26374]: mds.? [v2:172.21.15.38:6828/92551647,v1:172.21.15.38:6829/92551647] up:standby 2024-04-22T21:07:41.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:41 smithi038 ceph-mon[26374]: fsmap cephfs:1/1 {0=cephfs.smithi098.mipeta=up:reconnect} 3 up:standby 2024-04-22T21:07:41.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:41 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi038.vmivbd"}]: dispatch 2024-04-22T21:07:41.652 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:41 smithi098 ceph-mon[34053]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 614 B/s rd, 921 B/s wr, 3 op/s 2024-04-22T21:07:41.652 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:41 smithi098 ceph-mon[34053]: mds.? [v2:172.21.15.98:6824/2824268703,v1:172.21.15.98:6825/2824268703] up:reconnect 2024-04-22T21:07:41.652 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:41 smithi098 ceph-mon[34053]: mds.? [v2:172.21.15.38:6826/3829229281,v1:172.21.15.38:6827/3829229281] up:boot 2024-04-22T21:07:41.652 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:41 smithi098 ceph-mon[34053]: mds.? [v2:172.21.15.38:6828/92551647,v1:172.21.15.38:6829/92551647] up:standby 2024-04-22T21:07:41.652 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:41 smithi098 ceph-mon[34053]: fsmap cephfs:1/1 {0=cephfs.smithi098.mipeta=up:reconnect} 3 up:standby 2024-04-22T21:07:41.652 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:41 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi038.vmivbd"}]: dispatch 2024-04-22T21:07:42.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.558+0000 7f863a9f4700 1 -- 172.21.15.38:0/1003492176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86340fce50 msgr2=0x7f86340fd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:42.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.558+0000 7f863a9f4700 1 --2- 172.21.15.38:0/1003492176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86340fce50 0x7f86340fd260 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8624004940 tx=0x7f8624009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:42.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.558+0000 7f863a9f4700 1 -- 172.21.15.38:0/1003492176 shutdown_connections 2024-04-22T21:07:42.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.558+0000 7f863a9f4700 1 --2- 172.21.15.38:0/1003492176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86340fe050 0x7f86340fe4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:42.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.558+0000 7f863a9f4700 1 --2- 172.21.15.38:0/1003492176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86340fce50 0x7f86340fd260 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:42.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.558+0000 7f863a9f4700 1 -- 172.21.15.38:0/1003492176 >> 172.21.15.38:0/1003492176 conn(0x7f86340f8580 msgr2=0x7f86340fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:42.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.558+0000 7f863a9f4700 1 -- 172.21.15.38:0/1003492176 shutdown_connections 2024-04-22T21:07:42.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.558+0000 7f863a9f4700 1 -- 172.21.15.38:0/1003492176 wait complete. 2024-04-22T21:07:42.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.558+0000 7f863a9f4700 1 Processor -- start 2024-04-22T21:07:42.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.559+0000 7f863a9f4700 1 -- start start 2024-04-22T21:07:42.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.559+0000 7f863a9f4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86340fce50 0x7f863410dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:42.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.559+0000 7f863a9f4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86340fe050 0x7f863410dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:42.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.559+0000 7f863a9f4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863410e5f0 con 0x7f86340fe050 2024-04-22T21:07:42.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.559+0000 7f863a9f4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863410ee40 con 0x7f86340fce50 2024-04-22T21:07:42.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.559+0000 7f86337fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86340fe050 0x7f863410dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:42.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.559+0000 7f86337fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86340fe050 0x7f863410dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56054/0 (socket says 172.21.15.38:56054) 2024-04-22T21:07:42.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.559+0000 7f86337fe700 1 -- 172.21.15.38:0/978102244 learned_addr learned my addr 172.21.15.38:0/978102244 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:42.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.559+0000 7f8633fff700 1 --2- 172.21.15.38:0/978102244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86340fce50 0x7f863410dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:42.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.559+0000 7f86337fe700 1 -- 172.21.15.38:0/978102244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86340fce50 msgr2=0x7f863410dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:42.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.559+0000 7f86337fe700 1 --2- 172.21.15.38:0/978102244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86340fce50 0x7f863410dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:42.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.559+0000 7f86337fe700 1 -- 172.21.15.38:0/978102244 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8624005040 con 0x7f86340fe050 2024-04-22T21:07:42.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.559+0000 7f86337fe700 1 --2- 172.21.15.38:0/978102244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86340fe050 0x7f863410dff0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8628002a50 tx=0x7f86280046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:42.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.560+0000 7f8630ff9700 1 -- 172.21.15.38:0/978102244 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8628007cc0 con 0x7f86340fe050 2024-04-22T21:07:42.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.560+0000 7f8630ff9700 1 -- 172.21.15.38:0/978102244 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f862800f040 con 0x7f86340fe050 2024-04-22T21:07:42.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.560+0000 7f863a9f4700 1 -- 172.21.15.38:0/978102244 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f863410f120 con 0x7f86340fe050 2024-04-22T21:07:42.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.560+0000 7f8630ff9700 1 -- 172.21.15.38:0/978102244 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f862800da30 con 0x7f86340fe050 2024-04-22T21:07:42.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.560+0000 7f863a9f4700 1 -- 172.21.15.38:0/978102244 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f863410f660 con 0x7f86340fe050 2024-04-22T21:07:42.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.561+0000 7f863a9f4700 1 -- 172.21.15.38:0/978102244 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f863405f5f0 con 0x7f86340fe050 2024-04-22T21:07:42.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.561+0000 7f8630ff9700 1 -- 172.21.15.38:0/978102244 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f862800db90 con 0x7f86340fe050 2024-04-22T21:07:42.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.562+0000 7f8630ff9700 1 --2- 172.21.15.38:0/978102244 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f861c0692b0 0x7f861c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:42.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.562+0000 7f8633fff700 1 --2- 172.21.15.38:0/978102244 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f861c0692b0 0x7f861c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:42.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.562+0000 7f8630ff9700 1 -- 172.21.15.38:0/978102244 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f86280875d0 con 0x7f86340fe050 2024-04-22T21:07:42.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.562+0000 7f8633fff700 1 --2- 172.21.15.38:0/978102244 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f861c0692b0 0x7f861c06b760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8624000c00 tx=0x7f8624009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:42.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.566+0000 7f8630ff9700 1 -- 172.21.15.38:0/978102244 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f86280573f0 con 0x7f86340fe050 2024-04-22T21:07:42.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:42 smithi038 ceph-mon[26374]: mds.? [v2:172.21.15.98:6824/2824268703,v1:172.21.15.98:6825/2824268703] up:rejoin 2024-04-22T21:07:42.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:42 smithi038 ceph-mon[26374]: fsmap cephfs:1/1 {0=cephfs.smithi098.mipeta=up:rejoin} 3 up:standby 2024-04-22T21:07:42.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:42 smithi038 ceph-mon[26374]: daemon mds.cephfs.smithi098.mipeta is now active in filesystem cephfs as rank 0 2024-04-22T21:07:42.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:42 smithi098 ceph-mon[34053]: mds.? [v2:172.21.15.98:6824/2824268703,v1:172.21.15.98:6825/2824268703] up:rejoin 2024-04-22T21:07:42.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:42 smithi098 ceph-mon[34053]: fsmap cephfs:1/1 {0=cephfs.smithi098.mipeta=up:rejoin} 3 up:standby 2024-04-22T21:07:42.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:42 smithi098 ceph-mon[34053]: daemon mds.cephfs.smithi098.mipeta is now active in filesystem cephfs as rank 0 2024-04-22T21:07:42.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.822+0000 7f863a9f4700 1 -- 172.21.15.38:0/978102244 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f863404bed0 con 0x7f86340fe050 2024-04-22T21:07:42.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.822+0000 7f8630ff9700 1 -- 172.21.15.38:0/978102244 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1803 (secure 0 0 0) 0x7f8628056f80 con 0x7f86340fe050 2024-04-22T21:07:42.819 INFO:teuthology.orchestra.run.smithi038.stdout:e16 2024-04-22T21:07:42.819 INFO:teuthology.orchestra.run.smithi038.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T21:07:42.819 INFO:teuthology.orchestra.run.smithi038.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T21:07:42.819 INFO:teuthology.orchestra.run.smithi038.stdout:legacy client fscid: 1 2024-04-22T21:07:42.819 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:07:42.819 INFO:teuthology.orchestra.run.smithi038.stdout:Filesystem 'cephfs' (1) 2024-04-22T21:07:42.820 INFO:teuthology.orchestra.run.smithi038.stdout:fs_name cephfs 2024-04-22T21:07:42.820 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 16 2024-04-22T21:07:42.820 INFO:teuthology.orchestra.run.smithi038.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-22T21:07:42.820 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-04-22T21:07:25.224291+0000 2024-04-22T21:07:42.820 INFO:teuthology.orchestra.run.smithi038.stdout:modified 2024-04-22T21:07:42.335222+0000 2024-04-22T21:07:42.820 INFO:teuthology.orchestra.run.smithi038.stdout:tableserver 0 2024-04-22T21:07:42.820 INFO:teuthology.orchestra.run.smithi038.stdout:root 0 2024-04-22T21:07:42.820 INFO:teuthology.orchestra.run.smithi038.stdout:session_timeout 60 2024-04-22T21:07:42.820 INFO:teuthology.orchestra.run.smithi038.stdout:session_autoclose 300 2024-04-22T21:07:42.820 INFO:teuthology.orchestra.run.smithi038.stdout:max_file_size 1099511627776 2024-04-22T21:07:42.820 INFO:teuthology.orchestra.run.smithi038.stdout:required_client_features {} 2024-04-22T21:07:42.821 INFO:teuthology.orchestra.run.smithi038.stdout:last_failure 0 2024-04-22T21:07:42.821 INFO:teuthology.orchestra.run.smithi038.stdout:last_failure_osd_epoch 47 2024-04-22T21:07:42.821 INFO:teuthology.orchestra.run.smithi038.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T21:07:42.821 INFO:teuthology.orchestra.run.smithi038.stdout:max_mds 1 2024-04-22T21:07:42.821 INFO:teuthology.orchestra.run.smithi038.stdout:in 0 2024-04-22T21:07:42.821 INFO:teuthology.orchestra.run.smithi038.stdout:up {0=24279} 2024-04-22T21:07:42.821 INFO:teuthology.orchestra.run.smithi038.stdout:failed 2024-04-22T21:07:42.821 INFO:teuthology.orchestra.run.smithi038.stdout:damaged 2024-04-22T21:07:42.821 INFO:teuthology.orchestra.run.smithi038.stdout:stopped 2024-04-22T21:07:42.821 INFO:teuthology.orchestra.run.smithi038.stdout:data_pools [3] 2024-04-22T21:07:42.822 INFO:teuthology.orchestra.run.smithi038.stdout:metadata_pool 2 2024-04-22T21:07:42.822 INFO:teuthology.orchestra.run.smithi038.stdout:inline_data enabled 2024-04-22T21:07:42.822 INFO:teuthology.orchestra.run.smithi038.stdout:balancer 2024-04-22T21:07:42.822 INFO:teuthology.orchestra.run.smithi038.stdout:standby_count_wanted 1 2024-04-22T21:07:42.822 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.cephfs.smithi098.mipeta{0:24279} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.98:6824/2824268703,v1:172.21.15.98:6825/2824268703] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T21:07:42.822 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:07:42.822 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:07:42.822 INFO:teuthology.orchestra.run.smithi038.stdout:Standby daemons: 2024-04-22T21:07:42.822 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:07:42.822 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.cephfs.smithi038.hfegws{-1:14484} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.38:6828/92551647,v1:172.21.15.38:6829/92551647] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T21:07:42.822 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.cephfs.smithi038.vmivbd{-1:14496} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.38:6826/3829229281,v1:172.21.15.38:6827/3829229281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T21:07:42.822 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.cephfs.smithi098.izcoza{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.98:6826/1024898784,v1:172.21.15.98:6827/1024898784] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T21:07:42.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.825+0000 7f863a9f4700 1 -- 172.21.15.38:0/978102244 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f861c0692b0 msgr2=0x7f861c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:42.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.825+0000 7f863a9f4700 1 --2- 172.21.15.38:0/978102244 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f861c0692b0 0x7f861c06b760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8624000c00 tx=0x7f8624009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:42.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.825+0000 7f863a9f4700 1 -- 172.21.15.38:0/978102244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86340fe050 msgr2=0x7f863410dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:42.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.825+0000 7f863a9f4700 1 --2- 172.21.15.38:0/978102244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86340fe050 0x7f863410dff0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8628002a50 tx=0x7f86280046b0 comp rx=0 tx=0).stop 2024-04-22T21:07:42.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.826+0000 7f863a9f4700 1 -- 172.21.15.38:0/978102244 shutdown_connections 2024-04-22T21:07:42.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.826+0000 7f863a9f4700 1 --2- 172.21.15.38:0/978102244 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f861c0692b0 0x7f861c06b760 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:42.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.826+0000 7f863a9f4700 1 --2- 172.21.15.38:0/978102244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86340fce50 0x7f863410dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:42.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.826+0000 7f863a9f4700 1 --2- 172.21.15.38:0/978102244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86340fe050 0x7f863410dff0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:42.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.826+0000 7f863a9f4700 1 -- 172.21.15.38:0/978102244 >> 172.21.15.38:0/978102244 conn(0x7f86340f8580 msgr2=0x7f86340f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:42.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.826+0000 7f863a9f4700 1 -- 172.21.15.38:0/978102244 shutdown_connections 2024-04-22T21:07:42.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:42.826+0000 7f863a9f4700 1 -- 172.21.15.38:0/978102244 wait complete. 2024-04-22T21:07:42.825 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 16 2024-04-22T21:07:43.566 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:43 smithi038 ceph-mon[26374]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 819 B/s rd, 204 B/s wr, 1 op/s 2024-04-22T21:07:43.567 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:43 smithi038 ceph-mon[26374]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T21:07:43.567 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:43 smithi038 ceph-mon[26374]: mds.? [v2:172.21.15.98:6824/2824268703,v1:172.21.15.98:6825/2824268703] up:active 2024-04-22T21:07:43.567 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:43 smithi038 ceph-mon[26374]: mds.? [v2:172.21.15.98:6826/1024898784,v1:172.21.15.98:6827/1024898784] up:standby 2024-04-22T21:07:43.567 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:43 smithi038 ceph-mon[26374]: fsmap cephfs:1 {0=cephfs.smithi098.mipeta=up:active} 3 up:standby 2024-04-22T21:07:43.567 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:43 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/978102244' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T21:07:43.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:43 smithi098 ceph-mon[34053]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 819 B/s rd, 204 B/s wr, 1 op/s 2024-04-22T21:07:43.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:43 smithi098 ceph-mon[34053]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T21:07:43.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:43 smithi098 ceph-mon[34053]: mds.? [v2:172.21.15.98:6824/2824268703,v1:172.21.15.98:6825/2824268703] up:active 2024-04-22T21:07:43.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:43 smithi098 ceph-mon[34053]: mds.? [v2:172.21.15.98:6826/1024898784,v1:172.21.15.98:6827/1024898784] up:standby 2024-04-22T21:07:43.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:43 smithi098 ceph-mon[34053]: fsmap cephfs:1 {0=cephfs.smithi098.mipeta=up:active} 3 up:standby 2024-04-22T21:07:43.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:43 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/978102244' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T21:07:43.892 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-22T21:07:44.609 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:45.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:45 smithi038 ceph-mon[26374]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-22T21:07:45.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:45.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:45.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:07:45.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:07:45.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:45 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:45.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:45 smithi098 ceph-mon[34053]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-22T21:07:45.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:45 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:45.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:45 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:45.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:45 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:07:45.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:45 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:07:45.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:45 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:46.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.170+0000 7fe54eb02700 1 -- 172.21.15.38:0/2513552332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5480fff70 msgr2=0x7fe548102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:46.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.170+0000 7fe54eb02700 1 --2- 172.21.15.38:0/2513552332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5480fff70 0x7fe548102390 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe534004940 tx=0x7fe534009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:46.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.170+0000 7fe54eb02700 1 -- 172.21.15.38:0/2513552332 shutdown_connections 2024-04-22T21:07:46.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.170+0000 7fe54eb02700 1 --2- 172.21.15.38:0/2513552332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5480fff70 0x7fe548102390 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:46.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.170+0000 7fe54eb02700 1 --2- 172.21.15.38:0/2513552332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5480662f0 0x7fe5480ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.170+0000 7fe54eb02700 1 -- 172.21.15.38:0/2513552332 >> 172.21.15.38:0/2513552332 conn(0x7fe5480f74c0 msgr2=0x7fe5480f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.170+0000 7fe54eb02700 1 -- 172.21.15.38:0/2513552332 shutdown_connections 2024-04-22T21:07:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.170+0000 7fe54eb02700 1 -- 172.21.15.38:0/2513552332 wait complete. 2024-04-22T21:07:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.171+0000 7fe54eb02700 1 Processor -- start 2024-04-22T21:07:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.171+0000 7fe54eb02700 1 -- start start 2024-04-22T21:07:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.171+0000 7fe54eb02700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5480662f0 0x7fe5480fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.171+0000 7fe54eb02700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5480fff70 0x7fe5480fb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.171+0000 7fe54eb02700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5480fbee0 con 0x7fe5480662f0 2024-04-22T21:07:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.171+0000 7fe54eb02700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5480fc050 con 0x7fe5480fff70 2024-04-22T21:07:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.171+0000 7fe54c89e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5480662f0 0x7fe5480fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.171+0000 7fe54c89e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5480662f0 0x7fe5480fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49108/0 (socket says 172.21.15.38:49108) 2024-04-22T21:07:46.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.171+0000 7fe54c89e700 1 -- 172.21.15.38:0/2450934246 learned_addr learned my addr 172.21.15.38:0/2450934246 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:46.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.171+0000 7fe547fff700 1 --2- 172.21.15.38:0/2450934246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5480fff70 0x7fe5480fb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:46.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.172+0000 7fe54c89e700 1 -- 172.21.15.38:0/2450934246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5480fff70 msgr2=0x7fe5480fb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:46.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.172+0000 7fe54c89e700 1 --2- 172.21.15.38:0/2450934246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5480fff70 0x7fe5480fb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:46.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.172+0000 7fe54c89e700 1 -- 172.21.15.38:0/2450934246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe534005040 con 0x7fe5480662f0 2024-04-22T21:07:46.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.172+0000 7fe547fff700 1 --2- 172.21.15.38:0/2450934246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5480fff70 0x7fe5480fb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T21:07:46.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.172+0000 7fe54c89e700 1 --2- 172.21.15.38:0/2450934246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5480662f0 0x7fe5480fd460 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe53c002a50 tx=0x7fe53c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:46.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.172+0000 7fe5457fa700 1 -- 172.21.15.38:0/2450934246 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe53c007cc0 con 0x7fe5480662f0 2024-04-22T21:07:46.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.172+0000 7fe54eb02700 1 -- 172.21.15.38:0/2450934246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5480fc330 con 0x7fe5480662f0 2024-04-22T21:07:46.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.172+0000 7fe5457fa700 1 -- 172.21.15.38:0/2450934246 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe53c00f040 con 0x7fe5480662f0 2024-04-22T21:07:46.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.172+0000 7fe5457fa700 1 -- 172.21.15.38:0/2450934246 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe53c00d9c0 con 0x7fe5480662f0 2024-04-22T21:07:46.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.172+0000 7fe54eb02700 1 -- 172.21.15.38:0/2450934246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe548115a90 con 0x7fe5480662f0 2024-04-22T21:07:46.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.173+0000 7fe54eb02700 1 -- 172.21.15.38:0/2450934246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54805f5f0 con 0x7fe5480662f0 2024-04-22T21:07:46.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.174+0000 7fe5457fa700 1 -- 172.21.15.38:0/2450934246 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fe53c009730 con 0x7fe5480662f0 2024-04-22T21:07:46.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.174+0000 7fe5457fa700 1 --2- 172.21.15.38:0/2450934246 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe5380692b0 0x7fe53806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:46.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.174+0000 7fe5457fa700 1 -- 172.21.15.38:0/2450934246 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe53c086ef0 con 0x7fe5480662f0 2024-04-22T21:07:46.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.174+0000 7fe547fff700 1 --2- 172.21.15.38:0/2450934246 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe5380692b0 0x7fe53806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:46.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.175+0000 7fe547fff700 1 --2- 172.21.15.38:0/2450934246 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe5380692b0 0x7fe53806b760 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe53400b3b0 tx=0x7fe534009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:46.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.179+0000 7fe5457fa700 1 -- 172.21.15.38:0/2450934246 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe53c053150 con 0x7fe5480662f0 2024-04-22T21:07:46.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.430+0000 7fe54eb02700 1 -- 172.21.15.38:0/2450934246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe54804bed0 con 0x7fe5480662f0 2024-04-22T21:07:46.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.431+0000 7fe5457fa700 1 -- 172.21.15.38:0/2450934246 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4682 (secure 0 0 0) 0x7fe53c056770 con 0x7fe5480662f0 2024-04-22T21:07:46.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.433+0000 7fe54eb02700 1 -- 172.21.15.38:0/2450934246 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe5380692b0 msgr2=0x7fe53806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:46.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.433+0000 7fe54eb02700 1 --2- 172.21.15.38:0/2450934246 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe5380692b0 0x7fe53806b760 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe53400b3b0 tx=0x7fe534009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:46.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.433+0000 7fe54eb02700 1 -- 172.21.15.38:0/2450934246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5480662f0 msgr2=0x7fe5480fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:46.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.433+0000 7fe54eb02700 1 --2- 172.21.15.38:0/2450934246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5480662f0 0x7fe5480fd460 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe53c002a50 tx=0x7fe53c004600 comp rx=0 tx=0).stop 2024-04-22T21:07:46.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.433+0000 7fe54eb02700 1 -- 172.21.15.38:0/2450934246 shutdown_connections 2024-04-22T21:07:46.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.433+0000 7fe54eb02700 1 --2- 172.21.15.38:0/2450934246 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe5380692b0 0x7fe53806b760 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:46.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.433+0000 7fe54eb02700 1 --2- 172.21.15.38:0/2450934246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5480662f0 0x7fe5480fd460 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:46.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.433+0000 7fe54eb02700 1 --2- 172.21.15.38:0/2450934246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5480fff70 0x7fe5480fb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:46.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.433+0000 7fe54eb02700 1 -- 172.21.15.38:0/2450934246 >> 172.21.15.38:0/2450934246 conn(0x7fe5480f74c0 msgr2=0x7fe5480f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:46.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.433+0000 7fe54eb02700 1 -- 172.21.15.38:0/2450934246 shutdown_connections 2024-04-22T21:07:46.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:46.433+0000 7fe54eb02700 1 -- 172.21.15.38:0/2450934246 wait complete. 2024-04-22T21:07:46.430 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 16 2024-04-22T21:07:46.439 INFO:teuthology.orchestra.run.smithi038.stdout:true 2024-04-22T21:07:46.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:46 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:07:46.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:46 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:07:47.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:47 smithi098 ceph-mon[34053]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.9 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-22T21:07:47.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:47 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/2450934246' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T21:07:47.569 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:47 smithi038 ceph-mon[26374]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.9 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-22T21:07:47.569 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:47 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/2450934246' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T21:07:47.574 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-22T21:07:48.972 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:48.972 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:48 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:48.972 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:48 smithi038 ceph-mon[26374]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-22T21:07:49.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:49.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:48 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:49.000 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:48 smithi098 ceph-mon[34053]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-22T21:07:49.163 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:51.256 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:51 smithi038 ceph-mon[26374]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-22T21:07:51.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:51 smithi098 ceph-mon[34053]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-22T21:07:51.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.589+0000 7f85e47b5700 1 -- 172.21.15.38:0/277575289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85dc0fed90 msgr2=0x7f85dc0faed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.589+0000 7f85e47b5700 1 --2- 172.21.15.38:0/277575289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85dc0fed90 0x7f85dc0faed0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f85cc004940 tx=0x7f85cc009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.589+0000 7f85e47b5700 1 -- 172.21.15.38:0/277575289 shutdown_connections 2024-04-22T21:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.589+0000 7f85e47b5700 1 --2- 172.21.15.38:0/277575289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85dc0fed90 0x7f85dc0faed0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.589+0000 7f85e47b5700 1 --2- 172.21.15.38:0/277575289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85dc0fe440 0x7f85dc0fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.589+0000 7f85e47b5700 1 -- 172.21.15.38:0/277575289 >> 172.21.15.38:0/277575289 conn(0x7f85dc0f6be0 msgr2=0x7f85dc0f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.589+0000 7f85e47b5700 1 -- 172.21.15.38:0/277575289 shutdown_connections 2024-04-22T21:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.589+0000 7f85e47b5700 1 -- 172.21.15.38:0/277575289 wait complete. 2024-04-22T21:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.590+0000 7f85e47b5700 1 Processor -- start 2024-04-22T21:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.590+0000 7f85e47b5700 1 -- start start 2024-04-22T21:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.590+0000 7f85e47b5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85dc0fe440 0x7f85dc10d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.590+0000 7f85e47b5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85dc0fed90 0x7f85dc10de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.590+0000 7f85e47b5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc10e500 con 0x7f85dc0fed90 2024-04-22T21:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.590+0000 7f85e47b5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc112ea0 con 0x7f85dc0fe440 2024-04-22T21:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.590+0000 7f85e1d50700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85dc0fed90 0x7f85dc10de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.590+0000 7f85e1d50700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85dc0fed90 0x7f85dc10de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49118/0 (socket says 172.21.15.38:49118) 2024-04-22T21:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.590+0000 7f85e1d50700 1 -- 172.21.15.38:0/2409844398 learned_addr learned my addr 172.21.15.38:0/2409844398 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.590+0000 7f85e2551700 1 --2- 172.21.15.38:0/2409844398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85dc0fe440 0x7f85dc10d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.590+0000 7f85e1d50700 1 -- 172.21.15.38:0/2409844398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85dc0fe440 msgr2=0x7f85dc10d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.590+0000 7f85e1d50700 1 --2- 172.21.15.38:0/2409844398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85dc0fe440 0x7f85dc10d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.590+0000 7f85e1d50700 1 -- 172.21.15.38:0/2409844398 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85cc005040 con 0x7f85dc0fed90 2024-04-22T21:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.590+0000 7f85e2551700 1 --2- 172.21.15.38:0/2409844398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85dc0fe440 0x7f85dc10d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T21:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.590+0000 7f85e1d50700 1 --2- 172.21.15.38:0/2409844398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85dc0fed90 0x7f85dc10de70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f85cc009150 tx=0x7f85cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.590+0000 7f85d2ffd700 1 -- 172.21.15.38:0/2409844398 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85cc00e040 con 0x7f85dc0fed90 2024-04-22T21:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.591+0000 7f85d2ffd700 1 -- 172.21.15.38:0/2409844398 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85cc00bbf0 con 0x7f85dc0fed90 2024-04-22T21:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.591+0000 7f85e47b5700 1 -- 172.21.15.38:0/2409844398 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85dc1130a0 con 0x7f85dc0fed90 2024-04-22T21:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.591+0000 7f85d2ffd700 1 -- 172.21.15.38:0/2409844398 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85cc01c630 con 0x7f85dc0fed90 2024-04-22T21:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.591+0000 7f85e47b5700 1 -- 172.21.15.38:0/2409844398 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85dc113610 con 0x7f85dc0fed90 2024-04-22T21:07:51.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.591+0000 7f85e47b5700 1 -- 172.21.15.38:0/2409844398 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85dc04c710 con 0x7f85dc0fed90 2024-04-22T21:07:51.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.592+0000 7f85d2ffd700 1 -- 172.21.15.38:0/2409844398 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f85cc00bd60 con 0x7f85dc0fed90 2024-04-22T21:07:51.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.592+0000 7f85d2ffd700 1 --2- 172.21.15.38:0/2409844398 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f85c80692a0 0x7f85c806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:51.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.593+0000 7f85d2ffd700 1 -- 172.21.15.38:0/2409844398 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f85cc0885e0 con 0x7f85dc0fed90 2024-04-22T21:07:51.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.593+0000 7f85e2551700 1 --2- 172.21.15.38:0/2409844398 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f85c80692a0 0x7f85c806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:51.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.593+0000 7f85e2551700 1 --2- 172.21.15.38:0/2409844398 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f85c80692a0 0x7f85c806b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f85d8007fd0 tx=0x7f85d80074f0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:51.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.598+0000 7f85d2ffd700 1 -- 172.21.15.38:0/2409844398 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f85cc058330 con 0x7f85dc0fed90 2024-04-22T21:07:51.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.877+0000 7f85e47b5700 1 -- 172.21.15.38:0/2409844398 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f85dc105970 con 0x7f85dc0fed90 2024-04-22T21:07:51.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.877+0000 7f85d2ffd700 1 -- 172.21.15.38:0/2409844398 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v16) v1 ==== 78+0+99 (secure 0 0 0) 0x7f85cc054020 con 0x7f85dc0fed90 2024-04-22T21:07:51.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.879+0000 7f85e47b5700 1 -- 172.21.15.38:0/2409844398 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f85c80692a0 msgr2=0x7f85c806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:51.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.879+0000 7f85e47b5700 1 --2- 172.21.15.38:0/2409844398 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f85c80692a0 0x7f85c806b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f85d8007fd0 tx=0x7f85d80074f0 comp rx=0 tx=0).stop 2024-04-22T21:07:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.879+0000 7f85e47b5700 1 -- 172.21.15.38:0/2409844398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85dc0fed90 msgr2=0x7f85dc10de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.879+0000 7f85e47b5700 1 --2- 172.21.15.38:0/2409844398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85dc0fed90 0x7f85dc10de70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f85cc009150 tx=0x7f85cc009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.879+0000 7f85e47b5700 1 -- 172.21.15.38:0/2409844398 shutdown_connections 2024-04-22T21:07:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.879+0000 7f85e47b5700 1 --2- 172.21.15.38:0/2409844398 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f85c80692a0 0x7f85c806b750 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.879+0000 7f85e47b5700 1 --2- 172.21.15.38:0/2409844398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85dc0fe440 0x7f85dc10d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.879+0000 7f85e47b5700 1 --2- 172.21.15.38:0/2409844398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85dc0fed90 0x7f85dc10de70 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.879+0000 7f85e47b5700 1 -- 172.21.15.38:0/2409844398 >> 172.21.15.38:0/2409844398 conn(0x7f85dc0f6be0 msgr2=0x7f85dc0f81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.880+0000 7f85e47b5700 1 -- 172.21.15.38:0/2409844398 shutdown_connections 2024-04-22T21:07:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:51.880+0000 7f85e47b5700 1 -- 172.21.15.38:0/2409844398 wait complete. 2024-04-22T21:07:51.887 INFO:teuthology.orchestra.run.smithi038.stdout:true 2024-04-22T21:07:52.413 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:52 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/2409844398' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-22T21:07:52.440 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-22T21:07:52.455 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T21:07:52.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:52 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/2409844398' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-22T21:07:52.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.584+0000 7f3fed55d640 1 -- 172.21.15.38:0/122074765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe80fb0a0 msgr2=0x7f3fe80fb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:52.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.584+0000 7f3fed55d640 1 --2- 172.21.15.38:0/122074765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe80fb0a0 0x7f3fe80fb4b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3fd80048d0 tx=0x7f3fd8009da0 comp rx=0 tx=0).stop 2024-04-22T21:07:52.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.584+0000 7f3fed55d640 1 -- 172.21.15.38:0/122074765 shutdown_connections 2024-04-22T21:07:52.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.584+0000 7f3fed55d640 1 --2- 172.21.15.38:0/122074765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fe80fc370 0x7f3fe80fc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:52.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.584+0000 7f3fed55d640 1 --2- 172.21.15.38:0/122074765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe80fb0a0 0x7f3fe80fb4b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:52.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.584+0000 7f3fed55d640 1 -- 172.21.15.38:0/122074765 >> 172.21.15.38:0/122074765 conn(0x7f3fe80f6880 msgr2=0x7f3fe80f8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:52.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.584+0000 7f3fed55d640 1 -- 172.21.15.38:0/122074765 shutdown_connections 2024-04-22T21:07:52.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.584+0000 7f3fed55d640 1 -- 172.21.15.38:0/122074765 wait complete. 2024-04-22T21:07:52.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.585+0000 7f3fed55d640 1 Processor -- start 2024-04-22T21:07:52.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.585+0000 7f3fed55d640 1 -- start start 2024-04-22T21:07:52.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.585+0000 7f3fed55d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe80fb0a0 0x7f3fe810cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:52.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.585+0000 7f3fed55d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fe80fc370 0x7f3fe810d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:52.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.585+0000 7f3fed55d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe810d870 con 0x7f3fe80fb0a0 2024-04-22T21:07:52.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.585+0000 7f3fed55d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe81121e0 con 0x7f3fe80fc370 2024-04-22T21:07:52.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.585+0000 7f3fe6ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe80fb0a0 0x7f3fe810cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:52.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.585+0000 7f3fe6ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe80fb0a0 0x7f3fe810cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49140/0 (socket says 172.21.15.38:49140) 2024-04-22T21:07:52.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.585+0000 7f3fe6ffd640 1 -- 172.21.15.38:0/728555457 learned_addr learned my addr 172.21.15.38:0/728555457 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:52.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.585+0000 7f3fe67fc640 1 --2- 172.21.15.38:0/728555457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fe80fc370 0x7f3fe810d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:52.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.586+0000 7f3fe6ffd640 1 -- 172.21.15.38:0/728555457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fe80fc370 msgr2=0x7f3fe810d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:52.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.586+0000 7f3fe6ffd640 1 --2- 172.21.15.38:0/728555457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fe80fc370 0x7f3fe810d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:52.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.586+0000 7f3fe6ffd640 1 -- 172.21.15.38:0/728555457 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd8005040 con 0x7f3fe80fb0a0 2024-04-22T21:07:52.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.586+0000 7f3fe6ffd640 1 --2- 172.21.15.38:0/728555457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe80fb0a0 0x7f3fe810cce0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8007da0 tx=0x7f3fd8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:52.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.586+0000 7f3fcf7fe640 1 -- 172.21.15.38:0/728555457 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fd80094c0 con 0x7f3fe80fb0a0 2024-04-22T21:07:52.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.586+0000 7f3fcf7fe640 1 -- 172.21.15.38:0/728555457 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3fd8009620 con 0x7f3fe80fb0a0 2024-04-22T21:07:52.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.586+0000 7f3fed55d640 1 -- 172.21.15.38:0/728555457 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe8112320 con 0x7f3fe80fb0a0 2024-04-22T21:07:52.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.586+0000 7f3fcf7fe640 1 -- 172.21.15.38:0/728555457 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fd803b620 con 0x7f3fe80fb0a0 2024-04-22T21:07:52.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.586+0000 7f3fed55d640 1 -- 172.21.15.38:0/728555457 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe81127f0 con 0x7f3fe80fb0a0 2024-04-22T21:07:52.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.587+0000 7f3fed55d640 1 -- 172.21.15.38:0/728555457 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe8104440 con 0x7f3fe80fb0a0 2024-04-22T21:07:52.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.588+0000 7f3fcf7fe640 1 -- 172.21.15.38:0/728555457 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f3fd802f070 con 0x7f3fe80fb0a0 2024-04-22T21:07:52.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.588+0000 7f3fcf7fe640 1 --2- 172.21.15.38:0/728555457 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f3fd0069210 0x7f3fd006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:52.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.588+0000 7f3fcf7fe640 1 -- 172.21.15.38:0/728555457 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3fd80a97c0 con 0x7f3fe80fb0a0 2024-04-22T21:07:52.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.588+0000 7f3fe67fc640 1 --2- 172.21.15.38:0/728555457 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f3fd0069210 0x7f3fd006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:52.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.589+0000 7f3fe67fc640 1 --2- 172.21.15.38:0/728555457 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f3fd0069210 0x7f3fd006b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc009fd0 tx=0x7f3fdc009e50 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:52.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.593+0000 7f3fcf7fe640 1 -- 172.21.15.38:0/728555457 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f3fd8075a20 con 0x7f3fe80fb0a0 2024-04-22T21:07:52.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.792+0000 7f3fed55d640 1 -- 172.21.15.38:0/728555457 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fe804bcc0 con 0x7f3fe80fb0a0 2024-04-22T21:07:52.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.793+0000 7f3fcf7fe640 1 -- 172.21.15.38:0/728555457 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+14337 (secure 0 0 0) 0x7f3fd8079040 con 0x7f3fe80fb0a0 2024-04-22T21:07:52.789 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:07:52.789 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":47,"fsid":"3c5f5352-00eb-11ef-bc93-c7b262605968","created":"2024-04-22T21:00:25.280047+0000","modified":"2024-04-22T21:07:39.314152+0000","last_up_change":"2024-04-22T21:06:10.229585+0000","last_in_change":"2024-04-22T21:05:51.689258+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T21:04:56.557561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-22T21:07:23.883537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-22T21:07:24.650777+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeb7d8ce-e081-4a6a-8c2e-560bcb819a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":454874338},{"type":"v1","addr":"172.21.15.38:6803","nonce":454874338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":454874338},{"type":"v1","addr":"172.21.15.38:6805","nonce":454874338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":454874338},{"type":"v1","addr":"172.21.15.38:6809","nonce":454874338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":454874338},{"type":"v1","addr":"172.21.15.38:6807","nonce":454874338}]},"public_addr":"172.21.15.38:6803/454874338","cluster_addr":"172.21.15.38:6805/454874338","heartbeat_back_addr":"172.21.15.38:6809/454874338","heartbeat_front_addr":"172.21.15.38:6807/454874338","state":["exists","up"]},{"osd":1,"uuid":"7c90956f-6cb0-4a71-86c0-9c38e9710808","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1336148940},{"type":"v1","addr":"172.21.15.38:6811","nonce":1336148940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1336148940},{"type":"v1","addr":"172.21.15.38:6813","nonce":1336148940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1336148940},{"type":"v1","addr":"172.21.15.38:6817","nonce":1336148940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1336148940},{"type":"v1","addr":"172.21.15.38:6815","nonce":1336148940}]},"public_addr":"172.21.15.38:6811/1336148940","cluster_addr":"172.21.15.38:6813/1336148940","heartbeat_back_addr":"172.21.15.38:6817/1336148940","heartbeat_front_addr":"172.21.15.38:6815/1336148940","state":["exists","up"]},{"osd":2,"uuid":"379616b9-bdc7-4fd6-a1c5-b937e2ef3202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":47489714},{"type":"v1","addr":"172.21.15.38:6819","nonce":47489714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":47489714},{"type":"v1","addr":"172.21.15.38:6821","nonce":47489714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":47489714},{"type":"v1","addr":"172.21.15.38:6825","nonce":47489714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":47489714},{"type":"v1","addr":"172.21.15.38:6823","nonce":47489714}]},"public_addr":"172.21.15.38:6819/47489714","cluster_addr":"172.21.15.38:6821/47489714","heartbeat_back_addr":"172.21.15.38:6825/47489714","heartbeat_front_addr":"172.21.15.38:6823/47489714","state":["exists","up"]},{"osd":3,"uuid":"becb1602-d471-4c38-8cd1-781828c4b2f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2538219727},{"type":"v1","addr":"172.21.15.98:6801","nonce":2538219727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2538219727},{"type":"v1","addr":"172.21.15.98:6803","nonce":2538219727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2538219727},{"type":"v1","addr":"172.21.15.98:6807","nonce":2538219727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2538219727},{"type":"v1","addr":"172.21.15.98:6805","nonce":2538219727}]},"public_addr":"172.21.15.98:6801/2538219727","cluster_addr":"172.21.15.98:6803/2538219727","heartbeat_back_addr":"172.21.15.98:6807/2538219727","heartbeat_front_addr":"172.21.15.98:6805/2538219727","state":["exists","up"]},{"osd":4,"uuid":"c886eb15-63f1-4bb9-a229-0278c8372f5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3945015359},{"type":"v1","addr":"172.21.15.98:6809","nonce":3945015359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3945015359},{"type":"v1","addr":"172.21.15.98:6811","nonce":3945015359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3945015359},{"type":"v1","addr":"172.21.15.98:6815","nonce":3945015359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3945015359},{"type":"v1","addr":"172.21.15.98:6813","nonce":3945015359}]},"public_addr":"172.21.15.98:6809/3945015359","cluster_addr":"172.21.15.98:6811/3945015359","heartbeat_back_addr":"172.21.15.98:6815/3945015359","heartbeat_front_addr":"172.21.15.98:6813/3945015359","state":["exists","up"]},{"osd":5,"uuid":"58e5fcf4-d0cc-4498-ae06-d51d8f2d0933","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2442541230},{"type":"v1","addr":"172.21.15.98:6817","nonce":2442541230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2442541230},{"type":"v1","addr":"172.21.15.98:6819","nonce":2442541230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2442541230},{"type":"v1","addr":"172.21.15.98:6823","nonce":2442541230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2442541230},{"type":"v1","addr":"172.21.15.98:6821","nonce":2442541230}]},"public_addr":"172.21.15.98:6817/2442541230","cluster_addr":"172.21.15.98:6819/2442541230","heartbeat_back_addr":"172.21.15.98:6823/2442541230","heartbeat_front_addr":"172.21.15.98:6821/2442541230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:03:53.821993+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:04:24.335225+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:04:52.351848+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:05:17.370839+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:05:43.310131+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:06:09.074139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6827/3707300297":"2024-04-23T21:07:39.314124+0000","172.21.15.38:0/3342218679":"2024-04-23T21:00:59.025532+0000","172.21.15.38:0/4050800451":"2024-04-23T21:02:38.036748+0000","172.21.15.38:6800/2787760150":"2024-04-23T21:00:59.025532+0000","172.21.15.38:6801/2551561659":"2024-04-23T21:01:31.045112+0000","172.21.15.38:0/3765130717":"2024-04-23T21:00:59.025532+0000","172.21.15.38:6801/2787760150":"2024-04-23T21:00:59.025532+0000","172.21.15.38:6800/2551561659":"2024-04-23T21:01:31.045112+0000","172.21.15.38:0/2942485563":"2024-04-23T21:00:59.025532+0000","172.21.15.38:0/3486303720":"2024-04-23T21:01:31.045112+0000","172.21.15.38:6826/3707300297":"2024-04-23T21:07:39.314124+0000","172.21.15.38:0/2754521346":"2024-04-23T21:01:31.045112+0000","172.21.15.38:6800/989134174":"2024-04-23T21:02:38.036748+0000","172.21.15.38:0/2835639508":"2024-04-23T21:01:31.045112+0000","172.21.15.38:6801/989134174":"2024-04-23T21:02:38.036748+0000","172.21.15.38:0/420277838":"2024-04-23T21:02:38.036748+0000","172.21.15.38:0/3596300465":"2024-04-23T21:02:38.036748+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T21:07:52.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.795+0000 7f3fed55d640 1 -- 172.21.15.38:0/728555457 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f3fd0069210 msgr2=0x7f3fd006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:52.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.795+0000 7f3fed55d640 1 --2- 172.21.15.38:0/728555457 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f3fd0069210 0x7f3fd006b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc009fd0 tx=0x7f3fdc009e50 comp rx=0 tx=0).stop 2024-04-22T21:07:52.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.795+0000 7f3fed55d640 1 -- 172.21.15.38:0/728555457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe80fb0a0 msgr2=0x7f3fe810cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:52.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.795+0000 7f3fed55d640 1 --2- 172.21.15.38:0/728555457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe80fb0a0 0x7f3fe810cce0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8007da0 tx=0x7f3fd8009da0 comp rx=0 tx=0).stop 2024-04-22T21:07:52.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.795+0000 7f3fed55d640 1 -- 172.21.15.38:0/728555457 shutdown_connections 2024-04-22T21:07:52.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.795+0000 7f3fed55d640 1 --2- 172.21.15.38:0/728555457 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f3fd0069210 0x7f3fd006b6c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:52.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.795+0000 7f3fed55d640 1 --2- 172.21.15.38:0/728555457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fe80fc370 0x7f3fe810d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:52.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.795+0000 7f3fed55d640 1 --2- 172.21.15.38:0/728555457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe80fb0a0 0x7f3fe810cce0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:52.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.795+0000 7f3fed55d640 1 -- 172.21.15.38:0/728555457 >> 172.21.15.38:0/728555457 conn(0x7f3fe80f6880 msgr2=0x7f3fe80f8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:52.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.795+0000 7f3fed55d640 1 -- 172.21.15.38:0/728555457 shutdown_connections 2024-04-22T21:07:52.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.795+0000 7f3fed55d640 1 -- 172.21.15.38:0/728555457 wait complete. 2024-04-22T21:07:52.802 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T21:04:56.557561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-22T21:07:23.883537+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-22T21:07:24.650777+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T21:07:52.803 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T21:07:52.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.907+0000 7ff5e1625640 1 -- 172.21.15.38:0/2369973019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc066220 msgr2=0x7ff5dc0fedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:52.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.907+0000 7ff5e1625640 1 --2- 172.21.15.38:0/2369973019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc066220 0x7ff5dc0fedb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009c50 tx=0x7ff5d4004d10 comp rx=0 tx=0).stop 2024-04-22T21:07:52.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.907+0000 7ff5e1625640 1 -- 172.21.15.38:0/2369973019 shutdown_connections 2024-04-22T21:07:52.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.907+0000 7ff5e1625640 1 --2- 172.21.15.38:0/2369973019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5dc0ff2f0 0x7ff5dc1016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:52.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.907+0000 7ff5e1625640 1 --2- 172.21.15.38:0/2369973019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc066220 0x7ff5dc0fedb0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:52.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.907+0000 7ff5e1625640 1 -- 172.21.15.38:0/2369973019 >> 172.21.15.38:0/2369973019 conn(0x7ff5dc0f6880 msgr2=0x7ff5dc0f8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:52.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.907+0000 7ff5e1625640 1 -- 172.21.15.38:0/2369973019 shutdown_connections 2024-04-22T21:07:52.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.907+0000 7ff5e1625640 1 -- 172.21.15.38:0/2369973019 wait complete. 2024-04-22T21:07:52.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.907+0000 7ff5e1625640 1 Processor -- start 2024-04-22T21:07:52.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.907+0000 7ff5e1625640 1 -- start start 2024-04-22T21:07:52.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.908+0000 7ff5e1625640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc066220 0x7ff5dc0fc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:52.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.908+0000 7ff5e1625640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5dc0ff2f0 0x7ff5dc0facb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:52.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.908+0000 7ff5e1625640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0fcc30 con 0x7ff5dc066220 2024-04-22T21:07:52.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.908+0000 7ff5e1625640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0fb1f0 con 0x7ff5dc0ff2f0 2024-04-22T21:07:52.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.908+0000 7ff5daffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc066220 0x7ff5dc0fc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:52.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.908+0000 7ff5daffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc066220 0x7ff5dc0fc5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39168/0 (socket says 172.21.15.38:39168) 2024-04-22T21:07:52.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.908+0000 7ff5daffd640 1 -- 172.21.15.38:0/2322285969 learned_addr learned my addr 172.21.15.38:0/2322285969 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:52.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.908+0000 7ff5da7fc640 1 --2- 172.21.15.38:0/2322285969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5dc0ff2f0 0x7ff5dc0facb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:52.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.908+0000 7ff5daffd640 1 -- 172.21.15.38:0/2322285969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5dc0ff2f0 msgr2=0x7ff5dc0facb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:52.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.908+0000 7ff5daffd640 1 --2- 172.21.15.38:0/2322285969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5dc0ff2f0 0x7ff5dc0facb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:52.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.908+0000 7ff5daffd640 1 -- 172.21.15.38:0/2322285969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d40098b0 con 0x7ff5dc066220 2024-04-22T21:07:52.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.908+0000 7ff5daffd640 1 --2- 172.21.15.38:0/2322285969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc066220 0x7ff5dc0fc5e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4000f80 tx=0x7ff5d4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:52.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.908+0000 7ff5c37fe640 1 -- 172.21.15.38:0/2322285969 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5d403a070 con 0x7ff5dc066220 2024-04-22T21:07:52.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.909+0000 7ff5e1625640 1 -- 172.21.15.38:0/2322285969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5dc0fb470 con 0x7ff5dc066220 2024-04-22T21:07:52.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.909+0000 7ff5c37fe640 1 -- 172.21.15.38:0/2322285969 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff5d4035b20 con 0x7ff5dc066220 2024-04-22T21:07:52.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.909+0000 7ff5c37fe640 1 -- 172.21.15.38:0/2322285969 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5d403ebc0 con 0x7ff5dc066220 2024-04-22T21:07:52.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.909+0000 7ff5e1625640 1 -- 172.21.15.38:0/2322285969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5dc0fba10 con 0x7ff5dc066220 2024-04-22T21:07:52.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.910+0000 7ff5e1625640 1 -- 172.21.15.38:0/2322285969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5dc04bcc0 con 0x7ff5dc066220 2024-04-22T21:07:52.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.910+0000 7ff5c37fe640 1 -- 172.21.15.38:0/2322285969 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ff5d4032070 con 0x7ff5dc066220 2024-04-22T21:07:52.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.910+0000 7ff5c37fe640 1 --2- 172.21.15.38:0/2322285969 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff5c4069210 0x7ff5c406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:52.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.911+0000 7ff5c37fe640 1 -- 172.21.15.38:0/2322285969 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff5d40a9440 con 0x7ff5dc066220 2024-04-22T21:07:52.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.911+0000 7ff5da7fc640 1 --2- 172.21.15.38:0/2322285969 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff5c4069210 0x7ff5c406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:52.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.911+0000 7ff5da7fc640 1 --2- 172.21.15.38:0/2322285969 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff5c4069210 0x7ff5c406b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0007fd0 tx=0x7ff5d0009e90 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:52.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:52.915+0000 7ff5c37fe640 1 -- 172.21.15.38:0/2322285969 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff5d40756a0 con 0x7ff5dc066220 2024-04-22T21:07:53.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.089+0000 7ff5e1625640 1 -- 172.21.15.38:0/2322285969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff5dc114e60 con 0x7ff5dc066220 2024-04-22T21:07:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.089+0000 7ff5c37fe640 1 -- 172.21.15.38:0/2322285969 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff5d4078cc0 con 0x7ff5dc066220 2024-04-22T21:07:53.085 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T21:07:53.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.091+0000 7ff5e1625640 1 -- 172.21.15.38:0/2322285969 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff5c4069210 msgr2=0x7ff5c406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:53.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.091+0000 7ff5e1625640 1 --2- 172.21.15.38:0/2322285969 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff5c4069210 0x7ff5c406b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0007fd0 tx=0x7ff5d0009e90 comp rx=0 tx=0).stop 2024-04-22T21:07:53.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.091+0000 7ff5e1625640 1 -- 172.21.15.38:0/2322285969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc066220 msgr2=0x7ff5dc0fc5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:53.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.091+0000 7ff5e1625640 1 --2- 172.21.15.38:0/2322285969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc066220 0x7ff5dc0fc5e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4000f80 tx=0x7ff5d4004d10 comp rx=0 tx=0).stop 2024-04-22T21:07:53.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.091+0000 7ff5e1625640 1 -- 172.21.15.38:0/2322285969 shutdown_connections 2024-04-22T21:07:53.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.091+0000 7ff5e1625640 1 --2- 172.21.15.38:0/2322285969 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff5c4069210 0x7ff5c406b6c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.091+0000 7ff5e1625640 1 --2- 172.21.15.38:0/2322285969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5dc0ff2f0 0x7ff5dc0facb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.091+0000 7ff5e1625640 1 --2- 172.21.15.38:0/2322285969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc066220 0x7ff5dc0fc5e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.091+0000 7ff5e1625640 1 -- 172.21.15.38:0/2322285969 >> 172.21.15.38:0/2322285969 conn(0x7ff5dc0f6880 msgr2=0x7ff5dc0f6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:53.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.091+0000 7ff5e1625640 1 -- 172.21.15.38:0/2322285969 shutdown_connections 2024-04-22T21:07:53.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.091+0000 7ff5e1625640 1 -- 172.21.15.38:0/2322285969 wait complete. 2024-04-22T21:07:53.099 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-22T21:07:53.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.205+0000 7fdff4b5d640 1 -- 172.21.15.38:0/2591470759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdff00fd4a0 msgr2=0x7fdff00fd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:53.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.205+0000 7fdff4b5d640 1 --2- 172.21.15.38:0/2591470759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdff00fd4a0 0x7fdff00fd910 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40048d0 tx=0x7fdfe4009da0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.205+0000 7fdff4b5d640 1 -- 172.21.15.38:0/2591470759 shutdown_connections 2024-04-22T21:07:53.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.205+0000 7fdff4b5d640 1 --2- 172.21.15.38:0/2591470759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdff00fd4a0 0x7fdff00fd910 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.205+0000 7fdff4b5d640 1 --2- 172.21.15.38:0/2591470759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff00fc1d0 0x7fdff00fc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.205+0000 7fdff4b5d640 1 -- 172.21.15.38:0/2591470759 >> 172.21.15.38:0/2591470759 conn(0x7fdff00f7990 msgr2=0x7fdff00f9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:53.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.205+0000 7fdff4b5d640 1 -- 172.21.15.38:0/2591470759 shutdown_connections 2024-04-22T21:07:53.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.205+0000 7fdff4b5d640 1 -- 172.21.15.38:0/2591470759 wait complete. 2024-04-22T21:07:53.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.205+0000 7fdff4b5d640 1 Processor -- start 2024-04-22T21:07:53.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.205+0000 7fdff4b5d640 1 -- start start 2024-04-22T21:07:53.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.205+0000 7fdff4b5d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdff00fc1d0 0x7fdff010cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:53.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.205+0000 7fdff4b5d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff00fd4a0 0x7fdff010d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:53.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.205+0000 7fdff4b5d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff010d890 con 0x7fdff00fc1d0 2024-04-22T21:07:53.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.205+0000 7fdff4b5d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff0112250 con 0x7fdff00fd4a0 2024-04-22T21:07:53.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.207+0000 7fdfee59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdff00fc1d0 0x7fdff010cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:53.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.207+0000 7fdfee59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdff00fc1d0 0x7fdff010cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39194/0 (socket says 172.21.15.38:39194) 2024-04-22T21:07:53.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.207+0000 7fdfee59c640 1 -- 172.21.15.38:0/1125206383 learned_addr learned my addr 172.21.15.38:0/1125206383 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:53.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.207+0000 7fdfedd9b640 1 --2- 172.21.15.38:0/1125206383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff00fd4a0 0x7fdff010d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:53.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.207+0000 7fdfee59c640 1 -- 172.21.15.38:0/1125206383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff00fd4a0 msgr2=0x7fdff010d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:53.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.207+0000 7fdfee59c640 1 --2- 172.21.15.38:0/1125206383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff00fd4a0 0x7fdff010d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.207+0000 7fdfee59c640 1 -- 172.21.15.38:0/1125206383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe4005040 con 0x7fdff00fc1d0 2024-04-22T21:07:53.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.207+0000 7fdfee59c640 1 --2- 172.21.15.38:0/1125206383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdff00fc1d0 0x7fdff010cd00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8002a50 tx=0x7fdfe8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:53.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.207+0000 7fdfdeffd640 1 -- 172.21.15.38:0/1125206383 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfe8007c30 con 0x7fdff00fc1d0 2024-04-22T21:07:53.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.207+0000 7fdff4b5d640 1 -- 172.21.15.38:0/1125206383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff01123f0 con 0x7fdff00fc1d0 2024-04-22T21:07:53.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.207+0000 7fdfdeffd640 1 -- 172.21.15.38:0/1125206383 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdfe800f040 con 0x7fdff00fc1d0 2024-04-22T21:07:53.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.207+0000 7fdfdeffd640 1 -- 172.21.15.38:0/1125206383 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfe800db50 con 0x7fdff00fc1d0 2024-04-22T21:07:53.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.207+0000 7fdff4b5d640 1 -- 172.21.15.38:0/1125206383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff01128c0 con 0x7fdff00fc1d0 2024-04-22T21:07:53.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.207+0000 7fdff4b5d640 1 -- 172.21.15.38:0/1125206383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff004bcc0 con 0x7fdff00fc1d0 2024-04-22T21:07:53.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.208+0000 7fdfdeffd640 1 -- 172.21.15.38:0/1125206383 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fdfe801c400 con 0x7fdff00fc1d0 2024-04-22T21:07:53.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.208+0000 7fdfdeffd640 1 --2- 172.21.15.38:0/1125206383 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fdfd8069210 0x7fdfd806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:53.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.208+0000 7fdfdeffd640 1 -- 172.21.15.38:0/1125206383 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fdfe8086b20 con 0x7fdff00fc1d0 2024-04-22T21:07:53.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.208+0000 7fdfedd9b640 1 --2- 172.21.15.38:0/1125206383 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fdfd8069210 0x7fdfd806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:53.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.208+0000 7fdfedd9b640 1 --2- 172.21.15.38:0/1125206383 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fdfd8069210 0x7fdfd806b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40317e0 tx=0x7fdfe4009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:53.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.213+0000 7fdfdeffd640 1 -- 172.21.15.38:0/1125206383 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fdfe804d250 con 0x7fdff00fc1d0 2024-04-22T21:07:53.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:53 smithi038 ceph-mon[26374]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-22T21:07:53.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:53.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:53.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:07:53.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:07:53.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:53.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:53 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/728555457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:07:53.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:53 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/2322285969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T21:07:53.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:53.375 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:07:53.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.387+0000 7fdff4b5d640 1 -- 172.21.15.38:0/1125206383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fdff0104d40 con 0x7fdff00fc1d0 2024-04-22T21:07:53.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.387+0000 7fdfdeffd640 1 -- 172.21.15.38:0/1125206383 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7fdfe8052d80 con 0x7fdff00fc1d0 2024-04-22T21:07:53.382 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T21:07:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.389+0000 7fdff4b5d640 1 -- 172.21.15.38:0/1125206383 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fdfd8069210 msgr2=0x7fdfd806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.389+0000 7fdff4b5d640 1 --2- 172.21.15.38:0/1125206383 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fdfd8069210 0x7fdfd806b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40317e0 tx=0x7fdfe4009da0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.389+0000 7fdff4b5d640 1 -- 172.21.15.38:0/1125206383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdff00fc1d0 msgr2=0x7fdff010cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.389+0000 7fdff4b5d640 1 --2- 172.21.15.38:0/1125206383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdff00fc1d0 0x7fdff010cd00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8002a50 tx=0x7fdfe8004600 comp rx=0 tx=0).stop 2024-04-22T21:07:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.389+0000 7fdff4b5d640 1 -- 172.21.15.38:0/1125206383 shutdown_connections 2024-04-22T21:07:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.389+0000 7fdff4b5d640 1 --2- 172.21.15.38:0/1125206383 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fdfd8069210 0x7fdfd806b6c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.389+0000 7fdff4b5d640 1 --2- 172.21.15.38:0/1125206383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff00fd4a0 0x7fdff010d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.389+0000 7fdff4b5d640 1 --2- 172.21.15.38:0/1125206383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdff00fc1d0 0x7fdff010cd00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.389+0000 7fdff4b5d640 1 -- 172.21.15.38:0/1125206383 >> 172.21.15.38:0/1125206383 conn(0x7fdff00f7990 msgr2=0x7fdff00f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.389+0000 7fdff4b5d640 1 -- 172.21.15.38:0/1125206383 shutdown_connections 2024-04-22T21:07:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.389+0000 7fdff4b5d640 1 -- 172.21.15.38:0/1125206383 wait complete. 2024-04-22T21:07:53.395 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-22T21:07:53.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.501+0000 7fe52515d640 1 -- 172.21.15.38:0/1348325271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200faf70 msgr2=0x7fe5201017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:53.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.501+0000 7fe52515d640 1 --2- 172.21.15.38:0/1348325271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200faf70 0x7fe5201017c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe5080048d0 tx=0x7fe508009da0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.501+0000 7fe52515d640 1 -- 172.21.15.38:0/1348325271 shutdown_connections 2024-04-22T21:07:53.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.501+0000 7fe52515d640 1 --2- 172.21.15.38:0/1348325271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200faf70 0x7fe5201017c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.501+0000 7fe52515d640 1 --2- 172.21.15.38:0/1348325271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5200fa620 0x7fe5200faa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.501+0000 7fe52515d640 1 -- 172.21.15.38:0/1348325271 >> 172.21.15.38:0/1348325271 conn(0x7fe5200f6020 msgr2=0x7fe5200f8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:53.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.501+0000 7fe52515d640 1 -- 172.21.15.38:0/1348325271 shutdown_connections 2024-04-22T21:07:53.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.501+0000 7fe52515d640 1 -- 172.21.15.38:0/1348325271 wait complete. 2024-04-22T21:07:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.502+0000 7fe52515d640 1 Processor -- start 2024-04-22T21:07:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.502+0000 7fe52515d640 1 -- start start 2024-04-22T21:07:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.502+0000 7fe52515d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5200fa620 0x7fe5200fcf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.502+0000 7fe52515d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200faf70 0x7fe5200fb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.502+0000 7fe52515d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5200fd5a0 con 0x7fe5200faf70 2024-04-22T21:07:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.502+0000 7fe52515d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5200fbb60 con 0x7fe5200fa620 2024-04-22T21:07:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.502+0000 7fe517fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200faf70 0x7fe5200fb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.502+0000 7fe517fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200faf70 0x7fe5200fb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39208/0 (socket says 172.21.15.38:39208) 2024-04-22T21:07:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.502+0000 7fe517fff640 1 -- 172.21.15.38:0/2280746113 learned_addr learned my addr 172.21.15.38:0/2280746113 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.502+0000 7fe51ed9d640 1 --2- 172.21.15.38:0/2280746113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5200fa620 0x7fe5200fcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.502+0000 7fe517fff640 1 -- 172.21.15.38:0/2280746113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5200fa620 msgr2=0x7fe5200fcf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.502+0000 7fe517fff640 1 --2- 172.21.15.38:0/2280746113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5200fa620 0x7fe5200fcf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.502+0000 7fe517fff640 1 -- 172.21.15.38:0/2280746113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe508005040 con 0x7fe5200faf70 2024-04-22T21:07:53.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.502+0000 7fe517fff640 1 --2- 172.21.15.38:0/2280746113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200faf70 0x7fe5200fb620 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe50802bda0 tx=0x7fe508009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:53.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.503+0000 7fe5177fe640 1 -- 172.21.15.38:0/2280746113 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5080094c0 con 0x7fe5200faf70 2024-04-22T21:07:53.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.503+0000 7fe5177fe640 1 -- 172.21.15.38:0/2280746113 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe508009620 con 0x7fe5200faf70 2024-04-22T21:07:53.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.503+0000 7fe52515d640 1 -- 172.21.15.38:0/2280746113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5200fbde0 con 0x7fe5200faf70 2024-04-22T21:07:53.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.503+0000 7fe5177fe640 1 -- 172.21.15.38:0/2280746113 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe50803b640 con 0x7fe5200faf70 2024-04-22T21:07:53.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.503+0000 7fe52515d640 1 -- 172.21.15.38:0/2280746113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5200fc300 con 0x7fe5200faf70 2024-04-22T21:07:53.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:53 smithi098 ceph-mon[34053]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-22T21:07:53.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:53.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:53.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:07:53.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:07:53.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:53.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:53 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/728555457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:07:53.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:53 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/2322285969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T21:07:53.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:07:53.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:07:53.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.504+0000 7fe52515d640 1 -- 172.21.15.38:0/2280746113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe52004bcc0 con 0x7fe5200faf70 2024-04-22T21:07:53.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.504+0000 7fe5177fe640 1 -- 172.21.15.38:0/2280746113 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fe50802f070 con 0x7fe5200faf70 2024-04-22T21:07:53.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.505+0000 7fe5177fe640 1 --2- 172.21.15.38:0/2280746113 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe50c069210 0x7fe50c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:53.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.505+0000 7fe5177fe640 1 -- 172.21.15.38:0/2280746113 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe5080a96b0 con 0x7fe5200faf70 2024-04-22T21:07:53.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.505+0000 7fe51ed9d640 1 --2- 172.21.15.38:0/2280746113 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe50c069210 0x7fe50c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:53.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.505+0000 7fe51ed9d640 1 --2- 172.21.15.38:0/2280746113 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe50c069210 0x7fe50c06b6c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe510004ed0 tx=0x7fe51000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:53.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.509+0000 7fe5177fe640 1 -- 172.21.15.38:0/2280746113 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe508075990 con 0x7fe5200faf70 2024-04-22T21:07:53.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.683+0000 7fe52515d640 1 -- 172.21.15.38:0/2280746113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fe520104d70 con 0x7fe5200faf70 2024-04-22T21:07:53.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.683+0000 7fe5177fe640 1 -- 172.21.15.38:0/2280746113 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7fe508078fb0 con 0x7fe5200faf70 2024-04-22T21:07:53.678 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T21:07:53.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.685+0000 7fe52515d640 1 -- 172.21.15.38:0/2280746113 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe50c069210 msgr2=0x7fe50c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:53.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.685+0000 7fe52515d640 1 --2- 172.21.15.38:0/2280746113 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe50c069210 0x7fe50c06b6c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe510004ed0 tx=0x7fe51000b040 comp rx=0 tx=0).stop 2024-04-22T21:07:53.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.685+0000 7fe52515d640 1 -- 172.21.15.38:0/2280746113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200faf70 msgr2=0x7fe5200fb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:53.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.685+0000 7fe52515d640 1 --2- 172.21.15.38:0/2280746113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200faf70 0x7fe5200fb620 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe50802bda0 tx=0x7fe508009da0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.685+0000 7fe52515d640 1 -- 172.21.15.38:0/2280746113 shutdown_connections 2024-04-22T21:07:53.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.685+0000 7fe52515d640 1 --2- 172.21.15.38:0/2280746113 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe50c069210 0x7fe50c06b6c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.685+0000 7fe52515d640 1 --2- 172.21.15.38:0/2280746113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200faf70 0x7fe5200fb620 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.685+0000 7fe52515d640 1 --2- 172.21.15.38:0/2280746113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5200fa620 0x7fe5200fcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.685+0000 7fe52515d640 1 -- 172.21.15.38:0/2280746113 >> 172.21.15.38:0/2280746113 conn(0x7fe5200f6020 msgr2=0x7fe5200ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:53.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.685+0000 7fe52515d640 1 -- 172.21.15.38:0/2280746113 shutdown_connections 2024-04-22T21:07:53.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.685+0000 7fe52515d640 1 -- 172.21.15.38:0/2280746113 wait complete. 2024-04-22T21:07:53.693 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T21:07:53.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.795+0000 7fc1b3f1d640 1 -- 172.21.15.38:0/1211488569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1ac0fd8e0 msgr2=0x7fc1ac0fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:53.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.795+0000 7fc1b3f1d640 1 --2- 172.21.15.38:0/1211488569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1ac0fd8e0 0x7fc1ac0fdcf0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc1a40048d0 tx=0x7fc1a4009da0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.795+0000 7fc1b3f1d640 1 -- 172.21.15.38:0/1211488569 shutdown_connections 2024-04-22T21:07:53.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.795+0000 7fc1b3f1d640 1 --2- 172.21.15.38:0/1211488569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1ac0fe230 0x7fc1ac0fa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.795+0000 7fc1b3f1d640 1 --2- 172.21.15.38:0/1211488569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1ac0fd8e0 0x7fc1ac0fdcf0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.795+0000 7fc1b3f1d640 1 -- 172.21.15.38:0/1211488569 >> 172.21.15.38:0/1211488569 conn(0x7fc1ac0f6020 msgr2=0x7fc1ac0f8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:53.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.795+0000 7fc1b3f1d640 1 -- 172.21.15.38:0/1211488569 shutdown_connections 2024-04-22T21:07:53.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.795+0000 7fc1b3f1d640 1 -- 172.21.15.38:0/1211488569 wait complete. 2024-04-22T21:07:53.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.795+0000 7fc1b3f1d640 1 Processor -- start 2024-04-22T21:07:53.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.795+0000 7fc1b3f1d640 1 -- start start 2024-04-22T21:07:53.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1b3f1d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1ac0fd8e0 0x7fc1ac10cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1b3f1d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1ac0fe230 0x7fc1ac10d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1b3f1d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac10d9f0 con 0x7fc1ac0fd8e0 2024-04-22T21:07:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1b3f1d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac112320 con 0x7fc1ac0fe230 2024-04-22T21:07:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1b1cb9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1ac0fd8e0 0x7fc1ac10cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1b1cb9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1ac0fd8e0 0x7fc1ac10cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39220/0 (socket says 172.21.15.38:39220) 2024-04-22T21:07:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1b1cb9640 1 -- 172.21.15.38:0/3725625969 learned_addr learned my addr 172.21.15.38:0/3725625969 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1b14b8640 1 --2- 172.21.15.38:0/3725625969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1ac0fe230 0x7fc1ac10d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1b1cb9640 1 -- 172.21.15.38:0/3725625969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1ac0fe230 msgr2=0x7fc1ac10d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1b1cb9640 1 --2- 172.21.15.38:0/3725625969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1ac0fe230 0x7fc1ac10d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1b1cb9640 1 -- 172.21.15.38:0/3725625969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a4005040 con 0x7fc1ac0fd8e0 2024-04-22T21:07:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1b14b8640 1 --2- 172.21.15.38:0/3725625969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1ac0fe230 0x7fc1ac10d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T21:07:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1b1cb9640 1 --2- 172.21.15.38:0/3725625969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1ac0fd8e0 0x7fc1ac10cdd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4007d20 tx=0x7fc1a4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1aa7fc640 1 -- 172.21.15.38:0/3725625969 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1a4009590 con 0x7fc1ac0fd8e0 2024-04-22T21:07:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1b3f1d640 1 -- 172.21.15.38:0/3725625969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ac1124c0 con 0x7fc1ac0fd8e0 2024-04-22T21:07:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1aa7fc640 1 -- 172.21.15.38:0/3725625969 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1a40096f0 con 0x7fc1ac0fd8e0 2024-04-22T21:07:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1aa7fc640 1 -- 172.21.15.38:0/3725625969 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1a403b6d0 con 0x7fc1ac0fd8e0 2024-04-22T21:07:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.796+0000 7fc1b3f1d640 1 -- 172.21.15.38:0/3725625969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ac1129e0 con 0x7fc1ac0fd8e0 2024-04-22T21:07:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.797+0000 7fc1b3f1d640 1 -- 172.21.15.38:0/3725625969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ac04bcc0 con 0x7fc1ac0fd8e0 2024-04-22T21:07:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.798+0000 7fc1aa7fc640 1 -- 172.21.15.38:0/3725625969 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fc1a402f070 con 0x7fc1ac0fd8e0 2024-04-22T21:07:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.798+0000 7fc1aa7fc640 1 --2- 172.21.15.38:0/3725625969 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc194069210 0x7fc19406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.799+0000 7fc1b14b8640 1 --2- 172.21.15.38:0/3725625969 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc194069210 0x7fc19406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.799+0000 7fc1aa7fc640 1 -- 172.21.15.38:0/3725625969 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc1a40a98d0 con 0x7fc1ac0fd8e0 2024-04-22T21:07:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.799+0000 7fc1b14b8640 1 --2- 172.21.15.38:0/3725625969 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc194069210 0x7fc19406b6c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac0653d0 tx=0x7fc1a0008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:53.803+0000 7fc1aa7fc640 1 -- 172.21.15.38:0/3725625969 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fc1a4075b30 con 0x7fc1ac0fd8e0 2024-04-22T21:07:54.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:54.015+0000 7fc1b3f1d640 1 -- 172.21.15.38:0/3725625969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1ac112650 con 0x7fc1ac0fd8e0 2024-04-22T21:07:54.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:54.015+0000 7fc1aa7fc640 1 -- 172.21.15.38:0/3725625969 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4682 (secure 0 0 0) 0x7fc1ac112650 con 0x7fc1ac0fd8e0 2024-04-22T21:07:54.010 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T21:07:54.010 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"cephfs.smithi038.hfegws","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.38:6829/92551647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":92551647},{"type":"v1","addr":"172.21.15.38:6829","nonce":92551647}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":14496,"name":"cephfs.smithi038.vmivbd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6827/3829229281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3829229281},{"type":"v1","addr":"172.21.15.38:6827","nonce":3829229281}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24293,"name":"cephfs.smithi098.izcoza","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.98:6827/1024898784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1024898784},{"type":"v1","addr":"172.21.15.98:6827","nonce":1024898784}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T21:07:25.224291+0000","modified":"2024-04-22T21:07:42.335222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24279":{"gid":24279,"name":"cephfs.smithi098.mipeta","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6825/2824268703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2824268703},{"type":"v1","addr":"172.21.15.98:6825","nonce":2824268703}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-04-22T21:07:54.010 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 16 2024-04-22T21:07:54.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:54.017+0000 7fc1b3f1d640 1 -- 172.21.15.38:0/3725625969 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc194069210 msgr2=0x7fc19406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:54.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:54.017+0000 7fc1b3f1d640 1 --2- 172.21.15.38:0/3725625969 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc194069210 0x7fc19406b6c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac0653d0 tx=0x7fc1a0008040 comp rx=0 tx=0).stop 2024-04-22T21:07:54.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:54.017+0000 7fc1b3f1d640 1 -- 172.21.15.38:0/3725625969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1ac0fd8e0 msgr2=0x7fc1ac10cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:54.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:54.017+0000 7fc1b3f1d640 1 --2- 172.21.15.38:0/3725625969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1ac0fd8e0 0x7fc1ac10cdd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4007d20 tx=0x7fc1a4009da0 comp rx=0 tx=0).stop 2024-04-22T21:07:54.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:54.017+0000 7fc1b3f1d640 1 -- 172.21.15.38:0/3725625969 shutdown_connections 2024-04-22T21:07:54.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:54.017+0000 7fc1b3f1d640 1 --2- 172.21.15.38:0/3725625969 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fc194069210 0x7fc19406b6c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:54.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:54.017+0000 7fc1b3f1d640 1 --2- 172.21.15.38:0/3725625969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1ac0fe230 0x7fc1ac10d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:54.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:54.017+0000 7fc1b3f1d640 1 --2- 172.21.15.38:0/3725625969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1ac0fd8e0 0x7fc1ac10cdd0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:54.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:54.017+0000 7fc1b3f1d640 1 -- 172.21.15.38:0/3725625969 >> 172.21.15.38:0/3725625969 conn(0x7fc1ac0f6020 msgr2=0x7fc1ac0f76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:54.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:54.017+0000 7fc1b3f1d640 1 -- 172.21.15.38:0/3725625969 shutdown_connections 2024-04-22T21:07:54.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:54.017+0000 7fc1b3f1d640 1 -- 172.21.15.38:0/3725625969 wait complete. 2024-04-22T21:07:54.023 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 16, 'max_mds': 1, 'flags': 18} 2024-04-22T21:07:54.023 INFO:teuthology.run_tasks:Running task kclient... 2024-04-22T21:07:54.048 INFO:tasks.kclient:Mounting kernel clients... 2024-04-22T21:07:54.048 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-04-22T21:07:54.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:07:54.049 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list 2024-04-22T21:07:54.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:07:54.065 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link delete ceph-brx 2024-04-22T21:07:54.131 INFO:teuthology.orchestra.run.smithi038.stderr:Cannot find device "ceph-brx" 2024-04-22T21:07:54.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:07:54.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:07:54.133 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-22T21:07:54.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:07:54.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-04-22T21:07:54.224 INFO:teuthology.orchestra.run.smithi098.stderr:Cannot find device "ceph-brx" 2024-04-22T21:07:54.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:07:54.226 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-04-22T21:07:54.226 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T21:07:54.226 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T21:07:54.226 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-22T21:07:54.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:54 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/1125206383' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T21:07:54.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:54 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/2280746113' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T21:07:54.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:54 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3725625969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T21:07:54.489 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:54.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:54 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/1125206383' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T21:07:54.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:54 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/2280746113' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T21:07:54.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:54 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/3725625969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T21:07:55.428 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:55 smithi038 ceph-mon[26374]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-22T21:07:55.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:55 smithi098 ceph-mon[34053]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-22T21:07:55.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.896+0000 7f5283afa700 1 -- 172.21.15.38:0/3683263639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f527c1037c0 msgr2=0x7f527c0fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:55.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.896+0000 7f5283afa700 1 --2- 172.21.15.38:0/3683263639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f527c1037c0 0x7f527c0fce30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f526c004940 tx=0x7f526c009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:55.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.896+0000 7f5283afa700 1 -- 172.21.15.38:0/3683263639 shutdown_connections 2024-04-22T21:07:55.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.896+0000 7f5283afa700 1 --2- 172.21.15.38:0/3683263639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f527c1037c0 0x7f527c0fce30 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:55.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.896+0000 7f5283afa700 1 --2- 172.21.15.38:0/3683263639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f527c102e20 0x7f527c1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:55.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.896+0000 7f5283afa700 1 -- 172.21.15.38:0/3683263639 >> 172.21.15.38:0/3683263639 conn(0x7f527c0f89b0 msgr2=0x7f527c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:55.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.896+0000 7f5283afa700 1 -- 172.21.15.38:0/3683263639 shutdown_connections 2024-04-22T21:07:55.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.896+0000 7f5283afa700 1 -- 172.21.15.38:0/3683263639 wait complete. 2024-04-22T21:07:55.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.896+0000 7f5283afa700 1 Processor -- start 2024-04-22T21:07:55.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.897+0000 7f5283afa700 1 -- start start 2024-04-22T21:07:55.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.897+0000 7f5283afa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f527c102e20 0x7f527c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:55.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.897+0000 7f5283afa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f527c1037c0 0x7f527c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:55.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.897+0000 7f5283afa700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527c072210 con 0x7f527c1037c0 2024-04-22T21:07:55.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.897+0000 7f5283afa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527c070d40 con 0x7f527c102e20 2024-04-22T21:07:55.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.897+0000 7f5281095700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f527c1037c0 0x7f527c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:55.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.897+0000 7f5281095700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f527c1037c0 0x7f527c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39238/0 (socket says 172.21.15.38:39238) 2024-04-22T21:07:55.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.897+0000 7f5281095700 1 -- 172.21.15.38:0/2881419955 learned_addr learned my addr 172.21.15.38:0/2881419955 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:07:55.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.897+0000 7f5281896700 1 --2- 172.21.15.38:0/2881419955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f527c102e20 0x7f527c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:55.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.897+0000 7f5281095700 1 -- 172.21.15.38:0/2881419955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f527c102e20 msgr2=0x7f527c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:55.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.897+0000 7f5281095700 1 --2- 172.21.15.38:0/2881419955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f527c102e20 0x7f527c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:55.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.897+0000 7f5281095700 1 -- 172.21.15.38:0/2881419955 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526c005040 con 0x7f527c1037c0 2024-04-22T21:07:55.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.897+0000 7f5281095700 1 --2- 172.21.15.38:0/2881419955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f527c1037c0 0x7f527c070800 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f526c009100 tx=0x7f526c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:55.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.897+0000 7f52727fc700 1 -- 172.21.15.38:0/2881419955 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f526c00e040 con 0x7f527c1037c0 2024-04-22T21:07:55.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.897+0000 7f52727fc700 1 -- 172.21.15.38:0/2881419955 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f526c004290 con 0x7f527c1037c0 2024-04-22T21:07:55.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.898+0000 7f5283afa700 1 -- 172.21.15.38:0/2881419955 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527c070fc0 con 0x7f527c1037c0 2024-04-22T21:07:55.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.898+0000 7f52727fc700 1 -- 172.21.15.38:0/2881419955 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f526c01c3f0 con 0x7f527c1037c0 2024-04-22T21:07:55.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.898+0000 7f5283afa700 1 -- 172.21.15.38:0/2881419955 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f527c11a130 con 0x7f527c1037c0 2024-04-22T21:07:55.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.898+0000 7f5283afa700 1 -- 172.21.15.38:0/2881419955 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f527c1052f0 con 0x7f527c1037c0 2024-04-22T21:07:55.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.899+0000 7f52727fc700 1 -- 172.21.15.38:0/2881419955 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f526c013690 con 0x7f527c1037c0 2024-04-22T21:07:55.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.899+0000 7f52727fc700 1 --2- 172.21.15.38:0/2881419955 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f52680692a0 0x7f526806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:07:55.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.900+0000 7f52727fc700 1 -- 172.21.15.38:0/2881419955 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f526c088360 con 0x7f527c1037c0 2024-04-22T21:07:55.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.900+0000 7f5281896700 1 --2- 172.21.15.38:0/2881419955 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f52680692a0 0x7f526806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:07:55.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.900+0000 7f5281896700 1 --2- 172.21.15.38:0/2881419955 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f52680692a0 0x7f526806b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5278007fd0 tx=0x7f5278009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:07:55.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:55.904+0000 7f52727fc700 1 -- 172.21.15.38:0/2881419955 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f526c054640 con 0x7f527c1037c0 2024-04-22T21:07:56.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:56.148+0000 7f5283afa700 1 -- 172.21.15.38:0/2881419955 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f527c109f00 con 0x7f527c1037c0 2024-04-22T21:07:56.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:56.149+0000 7f52727fc700 1 -- 172.21.15.38:0/2881419955 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+83 (secure 0 0 0) 0x7f526c057c60 con 0x7f527c1037c0 2024-04-22T21:07:56.146 INFO:teuthology.orchestra.run.smithi038.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-22T21:07:56.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:56.150+0000 7f5283afa700 1 -- 172.21.15.38:0/2881419955 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f52680692a0 msgr2=0x7f526806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:56.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:56.150+0000 7f5283afa700 1 --2- 172.21.15.38:0/2881419955 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f52680692a0 0x7f526806b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5278007fd0 tx=0x7f5278009040 comp rx=0 tx=0).stop 2024-04-22T21:07:56.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:56.150+0000 7f5283afa700 1 -- 172.21.15.38:0/2881419955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f527c1037c0 msgr2=0x7f527c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:07:56.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:56.150+0000 7f5283afa700 1 --2- 172.21.15.38:0/2881419955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f527c1037c0 0x7f527c070800 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f526c009100 tx=0x7f526c009e30 comp rx=0 tx=0).stop 2024-04-22T21:07:56.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:56.151+0000 7f5283afa700 1 -- 172.21.15.38:0/2881419955 shutdown_connections 2024-04-22T21:07:56.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:56.151+0000 7f5283afa700 1 --2- 172.21.15.38:0/2881419955 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f52680692a0 0x7f526806b750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:56.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:56.151+0000 7f5283afa700 1 --2- 172.21.15.38:0/2881419955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f527c102e20 0x7f527c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:56.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:56.151+0000 7f5283afa700 1 --2- 172.21.15.38:0/2881419955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f527c1037c0 0x7f527c070800 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:07:56.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:56.151+0000 7f5283afa700 1 -- 172.21.15.38:0/2881419955 >> 172.21.15.38:0/2881419955 conn(0x7f527c0f89b0 msgr2=0x7f527c1021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:07:56.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:56.151+0000 7f5283afa700 1 -- 172.21.15.38:0/2881419955 shutdown_connections 2024-04-22T21:07:56.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:56.151+0000 7f5283afa700 1 -- 172.21.15.38:0/2881419955 wait complete. 2024-04-22T21:07:56.492 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:56 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/2881419955' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T21:07:56.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:56 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/2881419955' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T21:07:56.879 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T21:07:56.879 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T21:07:56.879 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi038.front.sepia.ceph.com 2024-04-22T21:07:56.880 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T21:07:56.880 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T21:07:56.880 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T21:07:56.880 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T21:07:56.880 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T21:07:56.880 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T21:07:56.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:07:56.880 DEBUG:teuthology.orchestra.run.smithi038:> ip addr 2024-04-22T21:07:56.897 INFO:teuthology.orchestra.run.smithi038.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T21:07:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T21:07:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T21:07:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-04-22T21:07:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: inet6 ::1/128 scope host 2024-04-22T21:07:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-04-22T21:07:56.898 INFO:teuthology.orchestra.run.smithi038.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T21:07:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:6b:fd:a2 brd ff:ff:ff:ff:ff:ff 2024-04-22T21:07:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp7s0f0 2024-04-22T21:07:56.898 INFO:teuthology.orchestra.run.smithi038.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T21:07:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:6b:fd:a3 brd ff:ff:ff:ff:ff:ff 2024-04-22T21:07:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp7s0f1 2024-04-22T21:07:56.898 INFO:teuthology.orchestra.run.smithi038.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T21:07:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:bd:15:fc brd ff:ff:ff:ff:ff:ff 2024-04-22T21:07:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp1s0f0 2024-04-22T21:07:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: inet 172.21.15.38/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-22T21:07:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft 42255sec preferred_lft 42255sec 2024-04-22T21:07:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: inet6 fe80::ec4:7aff:febd:15fc/64 scope link noprefixroute 2024-04-22T21:07:56.899 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-04-22T21:07:56.899 INFO:teuthology.orchestra.run.smithi038.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T21:07:56.899 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:bd:15:fd brd ff:ff:ff:ff:ff:ff 2024-04-22T21:07:56.899 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp1s0f1 2024-04-22T21:07:56.899 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T21:07:56.899 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T21:07:56.899 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T21:07:56.900 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link add name ceph-brx type bridge 2024-04-22T21:07:56.900 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip addr flush dev ceph-brx 2024-04-22T21:07:56.900 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set ceph-brx up 2024-04-22T21:07:56.900 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T21:07:56.900 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T21:07:56.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T21:07:57.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:07:57.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:07:57.082 DEBUG:teuthology.orchestra.run.smithi038:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T21:07:57.108 INFO:teuthology.orchestra.run.smithi038.stdout:1 2024-04-22T21:07:57.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:07:57.109 DEBUG:teuthology.orchestra.run.smithi038:> route 2024-04-22T21:07:57.185 INFO:teuthology.orchestra.run.smithi038.stdout:Kernel IP routing table 2024-04-22T21:07:57.185 INFO:teuthology.orchestra.run.smithi038.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T21:07:57.185 INFO:teuthology.orchestra.run.smithi038.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-22T21:07:57.185 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-22T21:07:57.185 INFO:teuthology.orchestra.run.smithi038.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T21:07:57.185 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T21:07:57.186 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T21:07:57.186 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-22T21:07:57.186 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-22T21:07:57.186 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-22T21:07:57.186 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T21:07:57.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T21:07:57.358 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:57 smithi038 ceph-mon[26374]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-22T21:07:57.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:07:57.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:07:57.449 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list 2024-04-22T21:07:57.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:07:57.465 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list-id 2024-04-22T21:07:57.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:57 smithi098 ceph-mon[34053]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-22T21:07:57.520 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T21:07:57.520 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T21:07:57.520 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T21:07:57.520 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-22T21:07:57.520 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T21:07:57.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T21:07:57.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:07:57.639 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-22T21:07:57.639 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T21:07:57.639 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T21:07:57.639 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-22T21:07:57.640 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T21:07:57.640 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-22T21:07:57.640 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-22T21:07:57.640 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-22T21:07:57.640 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T21:07:57.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T21:07:57.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:07:57.761 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T21:07:57.761 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T21:07:57.761 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set brx.0 up 2024-04-22T21:07:57.761 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T21:07:57.761 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T21:07:57.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T21:07:57.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:07:57.838 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T21:07:57.838 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T21:07:57.853 INFO:teuthology.orchestra.run.smithi038.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T21:07:57.853 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T21:07:57.853 DEBUG:teuthology.orchestra.run.smithi038:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T21:07:57.909 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-22T21:07:57.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:07:57.909 DEBUG:teuthology.orchestra.run.smithi038:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-22T21:07:57.992 INFO:teuthology.orchestra.run.smithi038.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-22T21:07:57.992 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-22T21:07:57.992 INFO:teuthology.orchestra.run.smithi038.stdout:invalid new device string format 2024-04-22T21:07:58.102 INFO:teuthology.orchestra.run.smithi038.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-22T21:07:58.102 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-22T21:07:58.102 INFO:teuthology.orchestra.run.smithi038.stdout:invalid new device string format 2024-04-22T21:07:58.102 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: resolved to: "172.21.15.38:3300,172.21.15.98:3300" 2024-04-22T21:07:58.102 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-22T21:07:58.107 INFO:teuthology.orchestra.run.smithi038.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-22T21:07:58.108 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-22T21:07:58.108 INFO:teuthology.orchestra.run.smithi038.stdout:invalid new device string format 2024-04-22T21:07:58.108 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: resolved to: "172.21.15.38:3300,172.21.15.98:3300" 2024-04-22T21:07:58.108 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-22T21:07:58.108 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: resolved to: "172.21.15.38:6789,172.21.15.98:6789" 2024-04-22T21:07:58.108 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.38:6789,172.21.15.98:6789:/ 2024-04-22T21:07:58.108 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=3c5f5352-00eb-11ef-bc93-c7b262605968" will pass to kernel 2024-04-22T21:07:58.109 INFO:teuthology.orchestra.run.smithi038.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-22T21:07:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: You just mounted a file system that supports labels which does not 2024-04-22T21:07:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-22T21:07:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: applications will generate AVC messages and not be allowed access to 2024-04-22T21:07:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-22T21:07:58.110 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-22T21:07:58.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:07:58.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T21:07:58.176 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T21:07:58.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T21:07:58.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:07:58.282 INFO:teuthology.orchestra.run.smithi038.stdout:client24327 2024-04-22T21:07:58.282 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T21:07:58.283 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T21:07:58.299 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T21:07:58.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T21:07:58.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:07:58.404 INFO:teuthology.orchestra.run.smithi038.stdout:client24327 2024-04-22T21:07:58.404 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T21:07:58.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T21:07:58.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:07:58.468 INFO:teuthology.orchestra.run.smithi038.stdout:client24327 2024-04-22T21:07:58.468 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968.client24327/status') 2024-04-22T21:07:58.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T21:07:58.532 INFO:teuthology.orchestra.run.smithi038.stdout:instance: client.24327 (3)172.21.15.38:0/2968269212 2024-04-22T21:07:58.533 INFO:teuthology.orchestra.run.smithi038.stdout:blocklisted: false 2024-04-22T21:07:58.533 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2024-04-22T21:07:58.533 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2024-04-22T21:07:58.533 INFO:teuthology.orchestra.run.smithi038.stderr:71 bytes copied, 2.5306e-05 s, 2.8 MB/s 2024-04-22T21:07:58.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:07:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:07:58.540 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-04-22T21:07:58.540 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T21:07:58.540 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T21:07:58.541 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-22T21:07:58.807 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:07:59.260 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:07:59 smithi038 ceph-mon[26374]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 0 op/s 2024-04-22T21:07:59.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:07:59 smithi098 ceph-mon[34053]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 0 op/s 2024-04-22T21:08:00.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.330+0000 7fdabedef700 1 -- 172.21.15.38:0/3413942877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdab81037c0 msgr2=0x7fdab80fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:00.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.330+0000 7fdabedef700 1 --2- 172.21.15.38:0/3413942877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdab81037c0 0x7fdab80fce30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4004940 tx=0x7fdaa4009e30 comp rx=0 tx=0).stop 2024-04-22T21:08:00.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.330+0000 7fdabedef700 1 -- 172.21.15.38:0/3413942877 shutdown_connections 2024-04-22T21:08:00.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.330+0000 7fdabedef700 1 --2- 172.21.15.38:0/3413942877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdab81037c0 0x7fdab80fce30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:00.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.330+0000 7fdabedef700 1 --2- 172.21.15.38:0/3413942877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab8102e20 0x7fdab81031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:00.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.330+0000 7fdabedef700 1 -- 172.21.15.38:0/3413942877 >> 172.21.15.38:0/3413942877 conn(0x7fdab80f89b0 msgr2=0x7fdab80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:08:00.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.330+0000 7fdabedef700 1 -- 172.21.15.38:0/3413942877 shutdown_connections 2024-04-22T21:08:00.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.330+0000 7fdabedef700 1 -- 172.21.15.38:0/3413942877 wait complete. 2024-04-22T21:08:00.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.331+0000 7fdabedef700 1 Processor -- start 2024-04-22T21:08:00.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.331+0000 7fdabedef700 1 -- start start 2024-04-22T21:08:00.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.331+0000 7fdabedef700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab8102e20 0x7fdab81139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:08:00.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.331+0000 7fdabedef700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdab81037c0 0x7fdab8113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:08:00.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.331+0000 7fdabedef700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab8114530 con 0x7fdab81037c0 2024-04-22T21:08:00.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.331+0000 7fdabedef700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab810db50 con 0x7fdab8102e20 2024-04-22T21:08:00.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.331+0000 7fdab7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdab81037c0 0x7fdab8113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:08:00.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.331+0000 7fdab7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdab81037c0 0x7fdab8113ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39260/0 (socket says 172.21.15.38:39260) 2024-04-22T21:08:00.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.331+0000 7fdab7fff700 1 -- 172.21.15.38:0/2570417368 learned_addr learned my addr 172.21.15.38:0/2570417368 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:08:00.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.331+0000 7fdabcb8b700 1 --2- 172.21.15.38:0/2570417368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab8102e20 0x7fdab81139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:08:00.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.332+0000 7fdab7fff700 1 -- 172.21.15.38:0/2570417368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab8102e20 msgr2=0x7fdab81139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:00.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.332+0000 7fdab7fff700 1 --2- 172.21.15.38:0/2570417368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab8102e20 0x7fdab81139a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:00.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.332+0000 7fdab7fff700 1 -- 172.21.15.38:0/2570417368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa4005040 con 0x7fdab81037c0 2024-04-22T21:08:00.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.332+0000 7fdab7fff700 1 --2- 172.21.15.38:0/2570417368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdab81037c0 0x7fdab8113ee0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4009100 tx=0x7fdaa4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:08:00.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.332+0000 7fdab57fa700 1 -- 172.21.15.38:0/2570417368 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdaa400e040 con 0x7fdab81037c0 2024-04-22T21:08:00.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.332+0000 7fdab57fa700 1 -- 172.21.15.38:0/2570417368 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdaa4009af0 con 0x7fdab81037c0 2024-04-22T21:08:00.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.332+0000 7fdabedef700 1 -- 172.21.15.38:0/2570417368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab810ddd0 con 0x7fdab81037c0 2024-04-22T21:08:00.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.332+0000 7fdab57fa700 1 -- 172.21.15.38:0/2570417368 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdaa401c3f0 con 0x7fdab81037c0 2024-04-22T21:08:00.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.332+0000 7fdabedef700 1 -- 172.21.15.38:0/2570417368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab810e390 con 0x7fdab81037c0 2024-04-22T21:08:00.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.333+0000 7fdabedef700 1 -- 172.21.15.38:0/2570417368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab81052f0 con 0x7fdab81037c0 2024-04-22T21:08:00.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.334+0000 7fdab57fa700 1 -- 172.21.15.38:0/2570417368 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fdaa400baf0 con 0x7fdab81037c0 2024-04-22T21:08:00.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.334+0000 7fdab57fa700 1 --2- 172.21.15.38:0/2570417368 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fdaa80692a0 0x7fdaa806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:08:00.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.334+0000 7fdab57fa700 1 -- 172.21.15.38:0/2570417368 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fdaa4088470 con 0x7fdab81037c0 2024-04-22T21:08:00.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.334+0000 7fdabcb8b700 1 --2- 172.21.15.38:0/2570417368 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fdaa80692a0 0x7fdaa806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:08:00.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.335+0000 7fdabcb8b700 1 --2- 172.21.15.38:0/2570417368 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fdaa80692a0 0x7fdaa806b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdaac004eb0 tx=0x7fdaac00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:08:00.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.339+0000 7fdab57fa700 1 -- 172.21.15.38:0/2570417368 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fdaa4054750 con 0x7fdab81037c0 2024-04-22T21:08:00.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.583+0000 7fdabedef700 1 -- 172.21.15.38:0/2570417368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdab810eff0 con 0x7fdab81037c0 2024-04-22T21:08:00.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.584+0000 7fdab57fa700 1 -- 172.21.15.38:0/2570417368 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+83 (secure 0 0 0) 0x7fdaa4057d70 con 0x7fdab81037c0 2024-04-22T21:08:00.580 INFO:teuthology.orchestra.run.smithi038.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-22T21:08:00.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.586+0000 7fdabedef700 1 -- 172.21.15.38:0/2570417368 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fdaa80692a0 msgr2=0x7fdaa806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:00.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.586+0000 7fdabedef700 1 --2- 172.21.15.38:0/2570417368 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fdaa80692a0 0x7fdaa806b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdaac004eb0 tx=0x7fdaac00b040 comp rx=0 tx=0).stop 2024-04-22T21:08:00.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.586+0000 7fdabedef700 1 -- 172.21.15.38:0/2570417368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdab81037c0 msgr2=0x7fdab8113ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:00.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.586+0000 7fdabedef700 1 --2- 172.21.15.38:0/2570417368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdab81037c0 0x7fdab8113ee0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4009100 tx=0x7fdaa4009e30 comp rx=0 tx=0).stop 2024-04-22T21:08:00.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.586+0000 7fdabedef700 1 -- 172.21.15.38:0/2570417368 shutdown_connections 2024-04-22T21:08:00.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.586+0000 7fdabedef700 1 --2- 172.21.15.38:0/2570417368 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fdaa80692a0 0x7fdaa806b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:00.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.586+0000 7fdabedef700 1 --2- 172.21.15.38:0/2570417368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdab8102e20 0x7fdab81139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:00.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.586+0000 7fdabedef700 1 --2- 172.21.15.38:0/2570417368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdab81037c0 0x7fdab8113ee0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:00.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.586+0000 7fdabedef700 1 -- 172.21.15.38:0/2570417368 >> 172.21.15.38:0/2570417368 conn(0x7fdab80f89b0 msgr2=0x7fdab81021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:08:00.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.586+0000 7fdabedef700 1 -- 172.21.15.38:0/2570417368 shutdown_connections 2024-04-22T21:08:00.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:00.586+0000 7fdabedef700 1 -- 172.21.15.38:0/2570417368 wait complete. 2024-04-22T21:08:01.201 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T21:08:01.201 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T21:08:01.201 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-04-22T21:08:01.201 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-22T21:08:01.201 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T21:08:01.201 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T21:08:01.201 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T21:08:01.201 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T21:08:01.202 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T21:08:01.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:08:01.202 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-04-22T21:08:01.221 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T21:08:01.221 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T21:08:01.221 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T21:08:01.221 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T21:08:01.221 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-04-22T21:08:01.221 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T21:08:01.222 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T21:08:01.222 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-04-22T21:08:01.222 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-04-22T21:08:01.222 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T21:08:01.222 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-04-22T21:08:01.222 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-04-22T21:08:01.222 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T21:08:01.222 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-04-22T21:08:01.222 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T21:08:01.222 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-04-22T21:08:01.222 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-22T21:08:01.222 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42260sec preferred_lft 42260sec 2024-04-22T21:08:01.222 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::bf6a:b9c2:e0ca:9593/64 scope link noprefixroute 2024-04-22T21:08:01.223 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-22T21:08:01.223 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T21:08:01.223 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T21:08:01.223 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T21:08:01.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2024-04-22T21:08:01.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2024-04-22T21:08:01.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2024-04-22T21:08:01.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T21:08:01.223 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T21:08:01.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T21:08:01.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:01 smithi038 ceph-mon[26374]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-22T21:08:01.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:01 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/2570417368' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T21:08:01.478 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:01 smithi098 ceph-mon[34053]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-22T21:08:01.479 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:01 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/2570417368' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T21:08:01.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:08:01.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:08:01.545 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T21:08:01.569 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-04-22T21:08:01.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:08:01.571 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-04-22T21:08:01.637 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-04-22T21:08:01.638 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T21:08:01.638 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T21:08:01.638 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T21:08:01.638 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T21:08:01.638 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T21:08:01.638 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T21:08:01.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T21:08:01.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T21:08:01.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T21:08:01.639 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T21:08:01.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T21:08:01.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:08:01.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:08:01.935 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-22T21:08:01.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:08:01.950 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2024-04-22T21:08:02.005 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T21:08:02.005 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T21:08:02.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T21:08:02.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-22T21:08:02.005 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T21:08:02.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T21:08:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:08:02.122 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-22T21:08:02.122 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T21:08:02.122 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T21:08:02.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-22T21:08:02.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T21:08:02.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-22T21:08:02.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-22T21:08:02.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-22T21:08:02.123 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T21:08:02.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T21:08:02.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:08:02.255 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T21:08:02.255 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-22T21:08:02.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-04-22T21:08:02.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T21:08:02.255 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-22T21:08:02.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T21:08:02.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:08:02.328 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T21:08:02.328 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-22T21:08:02.342 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-22T21:08:02.342 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T21:08:02.342 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-22T21:08:02.396 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-22T21:08:02.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:08:02.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-22T21:08:02.494 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-22T21:08:02.494 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-22T21:08:02.494 INFO:teuthology.orchestra.run.smithi098.stdout:invalid new device string format 2024-04-22T21:08:02.595 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-22T21:08:02.596 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-22T21:08:02.596 INFO:teuthology.orchestra.run.smithi098.stdout:invalid new device string format 2024-04-22T21:08:02.596 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.38:3300,172.21.15.98:3300" 2024-04-22T21:08:02.596 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-22T21:08:02.600 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-22T21:08:02.600 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-22T21:08:02.600 INFO:teuthology.orchestra.run.smithi098.stdout:invalid new device string format 2024-04-22T21:08:02.600 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.38:3300,172.21.15.98:3300" 2024-04-22T21:08:02.600 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-22T21:08:02.600 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.38:6789,172.21.15.98:6789" 2024-04-22T21:08:02.601 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.38:6789,172.21.15.98:6789:/ 2024-04-22T21:08:02.601 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=3c5f5352-00eb-11ef-bc93-c7b262605968" will pass to kernel 2024-04-22T21:08:02.604 INFO:teuthology.orchestra.run.smithi098.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-22T21:08:02.604 INFO:teuthology.orchestra.run.smithi098.stdout: You just mounted a file system that supports labels which does not 2024-04-22T21:08:02.604 INFO:teuthology.orchestra.run.smithi098.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-22T21:08:02.604 INFO:teuthology.orchestra.run.smithi098.stdout: applications will generate AVC messages and not be allowed access to 2024-04-22T21:08:02.604 INFO:teuthology.orchestra.run.smithi098.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-22T21:08:02.606 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-22T21:08:02.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T21:08:02.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-22T21:08:02.629 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T21:08:02.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T21:08:02.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:08:02.725 INFO:teuthology.orchestra.run.smithi098.stdout:client14538 2024-04-22T21:08:02.726 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T21:08:02.726 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T21:08:02.742 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T21:08:02.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T21:08:02.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:08:02.836 INFO:teuthology.orchestra.run.smithi098.stdout:client14538 2024-04-22T21:08:02.837 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T21:08:02.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T21:08:02.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:08:02.891 INFO:teuthology.orchestra.run.smithi098.stdout:client14538 2024-04-22T21:08:02.891 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968.client14538/status') 2024-04-22T21:08:02.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T21:08:02.948 INFO:teuthology.orchestra.run.smithi098.stdout:instance: client.14538 (3)172.21.15.98:0/4152540236 2024-04-22T21:08:02.948 INFO:teuthology.orchestra.run.smithi098.stdout:blocklisted: false 2024-04-22T21:08:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records in 2024-04-22T21:08:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records out 2024-04-22T21:08:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:71 bytes copied, 2.1706e-05 s, 3.3 MB/s 2024-04-22T21:08:02.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-22T21:08:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T21:08:02.955 INFO:teuthology.run_tasks:Running task print... 2024-04-22T21:08:02.967 INFO:teuthology.task.print:**** done client 2024-04-22T21:08:02.968 INFO:teuthology.run_tasks:Running task parallel... 2024-04-22T21:08:02.977 INFO:teuthology.task.parallel:starting parallel... 2024-04-22T21:08:02.977 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-22T21:08:02.978 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T21:08:02.978 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T21:08:02.979 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-22T21:08:02.979 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-22T21:08:02.979 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-04-22T21:08:02.983 INFO:tasks.workunit:Pulling workunits from ref 8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 2024-04-22T21:08:02.983 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-22T21:08:02.983 INFO:tasks.workunit:timeout=3h 2024-04-22T21:08:02.983 INFO:tasks.workunit:cleanup=True 2024-04-22T21:08:02.984 DEBUG:teuthology.orchestra.run.smithi038:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-22T21:08:03.005 INFO:teuthology.orchestra.run.smithi038.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-22T21:08:03.005 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-22T21:08:03.005 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 86h/134d Inode: 1 Links: 2 2024-04-22T21:08:03.005 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:08:03.005 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-22T21:08:03.006 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-22T21:08:03.006 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-22 21:07:29.250449593 +0000 2024-04-22T21:08:03.006 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-22 21:07:58.179005751 +0000 2024-04-22T21:08:03.006 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-22 21:07:29.250449593 +0000 2024-04-22T21:08:03.006 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-22T21:08:03.006 DEBUG:teuthology.orchestra.run.smithi038:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-22T21:08:03.076 DEBUG:teuthology.orchestra.run.smithi098:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-22T21:08:03.091 INFO:teuthology.orchestra.run.smithi098.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-04-22T21:08:03.092 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-04-22T21:08:03.092 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 70h/112d Inode: 1 Links: 3 2024-04-22T21:08:03.092 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:08:03.092 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-22T21:08:03.092 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-22T21:08:03.092 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-22 21:08:03.075918051 +0000 2024-04-22T21:08:03.092 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-22 21:08:03.075918051 +0000 2024-04-22T21:08:03.092 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-22 21:07:29.250449593 +0000 2024-04-22T21:08:03.092 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-04-22T21:08:03.093 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-04-22T21:08:03.159 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 2024-04-22T21:08:03.159 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 2024-04-22T21:08:03.175 INFO:tasks.workunit.client.0.smithi038.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-22T21:08:03.214 INFO:tasks.workunit.client.1.smithi098.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-22T21:08:03.264 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:08:03.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:03 smithi038 ceph-mon[26374]: pgmap v197: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-22T21:08:03.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:03 smithi098 ceph-mon[34053]: pgmap v197: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-22T21:08:04.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.777+0000 7f1615736700 1 -- 172.21.15.38:0/2722688413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16100fbcf0 msgr2=0x7f16100fc100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:04.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.777+0000 7f1615736700 1 --2- 172.21.15.38:0/2722688413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16100fbcf0 0x7f16100fc100 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1600004940 tx=0x7f1600009e30 comp rx=0 tx=0).stop 2024-04-22T21:08:04.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.777+0000 7f1615736700 1 -- 172.21.15.38:0/2722688413 shutdown_connections 2024-04-22T21:08:04.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.777+0000 7f1615736700 1 --2- 172.21.15.38:0/2722688413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16100fcfa0 0x7f16100fd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:04.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.777+0000 7f1615736700 1 --2- 172.21.15.38:0/2722688413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16100fbcf0 0x7f16100fc100 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:04.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.777+0000 7f1615736700 1 -- 172.21.15.38:0/2722688413 >> 172.21.15.38:0/2722688413 conn(0x7f16100f7450 msgr2=0x7f16100f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:08:04.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.777+0000 7f1615736700 1 -- 172.21.15.38:0/2722688413 shutdown_connections 2024-04-22T21:08:04.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.777+0000 7f1615736700 1 -- 172.21.15.38:0/2722688413 wait complete. 2024-04-22T21:08:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.777+0000 7f1615736700 1 Processor -- start 2024-04-22T21:08:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.778+0000 7f1615736700 1 -- start start 2024-04-22T21:08:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.778+0000 7f1615736700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16100fbcf0 0x7f1610066ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:08:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.778+0000 7f1615736700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16100fcfa0 0x7f1610112700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:08:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.778+0000 7f1615736700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1610112cd0 con 0x7f16100fbcf0 2024-04-22T21:08:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.778+0000 7f1615736700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1610112e10 con 0x7f16100fcfa0 2024-04-22T21:08:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.778+0000 7f160effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16100fbcf0 0x7f1610066ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:08:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.778+0000 7f160effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16100fbcf0 0x7f1610066ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41856/0 (socket says 172.21.15.38:41856) 2024-04-22T21:08:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.778+0000 7f160effd700 1 -- 172.21.15.38:0/1018518912 learned_addr learned my addr 172.21.15.38:0/1018518912 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:08:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.778+0000 7f160e7fc700 1 --2- 172.21.15.38:0/1018518912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16100fcfa0 0x7f1610112700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:08:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.778+0000 7f160effd700 1 -- 172.21.15.38:0/1018518912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16100fcfa0 msgr2=0x7f1610112700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:04.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.778+0000 7f160effd700 1 --2- 172.21.15.38:0/1018518912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16100fcfa0 0x7f1610112700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:04.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.778+0000 7f160effd700 1 -- 172.21.15.38:0/1018518912 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1600005040 con 0x7f16100fbcf0 2024-04-22T21:08:04.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.778+0000 7f160effd700 1 --2- 172.21.15.38:0/1018518912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16100fbcf0 0x7f1610066ce0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f160000b3b0 tx=0x7f1600009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:08:04.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.778+0000 7f15f77fe700 1 -- 172.21.15.38:0/1018518912 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f160000e040 con 0x7f16100fbcf0 2024-04-22T21:08:04.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.779+0000 7f1615736700 1 -- 172.21.15.38:0/1018518912 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1610113090 con 0x7f16100fbcf0 2024-04-22T21:08:04.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.779+0000 7f15f77fe700 1 -- 172.21.15.38:0/1018518912 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f160000bcc0 con 0x7f16100fbcf0 2024-04-22T21:08:04.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.779+0000 7f15f77fe700 1 -- 172.21.15.38:0/1018518912 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f160001c630 con 0x7f16100fbcf0 2024-04-22T21:08:04.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.779+0000 7f1615736700 1 -- 172.21.15.38:0/1018518912 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1610113650 con 0x7f16100fbcf0 2024-04-22T21:08:04.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.780+0000 7f1615736700 1 -- 172.21.15.38:0/1018518912 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f161004c710 con 0x7f16100fbcf0 2024-04-22T21:08:04.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.780+0000 7f15f77fe700 1 -- 172.21.15.38:0/1018518912 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f1600013980 con 0x7f16100fbcf0 2024-04-22T21:08:04.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.780+0000 7f15f77fe700 1 --2- 172.21.15.38:0/1018518912 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f15f80692a0 0x7f15f806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:08:04.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.781+0000 7f15f77fe700 1 -- 172.21.15.38:0/1018518912 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1600088020 con 0x7f16100fbcf0 2024-04-22T21:08:04.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.781+0000 7f160e7fc700 1 --2- 172.21.15.38:0/1018518912 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f15f80692a0 0x7f15f806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:08:04.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.781+0000 7f160e7fc700 1 --2- 172.21.15.38:0/1018518912 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f15f80692a0 0x7f15f806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1604006fd0 tx=0x7f1604008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:08:04.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.785+0000 7f15f77fe700 1 -- 172.21.15.38:0/1018518912 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1600057d10 con 0x7f16100fbcf0 2024-04-22T21:08:04.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.980+0000 7f1615736700 1 -- 172.21.15.38:0/1018518912 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f1610104860 con 0x7f16100fbcf0 2024-04-22T21:08:04.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.980+0000 7f15f77fe700 1 -- 172.21.15.38:0/1018518912 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f16000578a0 con 0x7f16100fbcf0 2024-04-22T21:08:04.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.982+0000 7f1615736700 1 -- 172.21.15.38:0/1018518912 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f15f80692a0 msgr2=0x7f15f806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:04.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.982+0000 7f1615736700 1 --2- 172.21.15.38:0/1018518912 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f15f80692a0 0x7f15f806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1604006fd0 tx=0x7f1604008040 comp rx=0 tx=0).stop 2024-04-22T21:08:04.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.982+0000 7f1615736700 1 -- 172.21.15.38:0/1018518912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16100fbcf0 msgr2=0x7f1610066ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:04.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.982+0000 7f1615736700 1 --2- 172.21.15.38:0/1018518912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16100fbcf0 0x7f1610066ce0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f160000b3b0 tx=0x7f1600009e30 comp rx=0 tx=0).stop 2024-04-22T21:08:04.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.982+0000 7f1615736700 1 -- 172.21.15.38:0/1018518912 shutdown_connections 2024-04-22T21:08:04.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.982+0000 7f1615736700 1 --2- 172.21.15.38:0/1018518912 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f15f80692a0 0x7f15f806b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:04.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.982+0000 7f1615736700 1 --2- 172.21.15.38:0/1018518912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16100fbcf0 0x7f1610066ce0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:04.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.982+0000 7f1615736700 1 --2- 172.21.15.38:0/1018518912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16100fcfa0 0x7f1610112700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:04.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.982+0000 7f1615736700 1 -- 172.21.15.38:0/1018518912 >> 172.21.15.38:0/1018518912 conn(0x7f16100f7450 msgr2=0x7f16100f8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:08:04.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.982+0000 7f1615736700 1 -- 172.21.15.38:0/1018518912 shutdown_connections 2024-04-22T21:08:04.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:04.982+0000 7f1615736700 1 -- 172.21.15.38:0/1018518912 wait complete. 2024-04-22T21:08:05.374 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:05 smithi038 ceph-mon[26374]: pgmap v198: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-22T21:08:05.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:05 smithi098 ceph-mon[34053]: pgmap v198: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-22T21:08:05.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-22T21:08:05.943 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:08:07.255 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:07 smithi038 ceph-mon[26374]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-04-22T21:08:07.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.482+0000 7f20f9d5d700 1 -- 172.21.15.38:0/4207581114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f4066f80 msgr2=0x7f20f40ff9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:07.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.482+0000 7f20f9d5d700 1 --2- 172.21.15.38:0/4207581114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f4066f80 0x7f20f40ff9f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f20e4004940 tx=0x7f20e4009e30 comp rx=0 tx=0).stop 2024-04-22T21:08:07.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.482+0000 7f20f9d5d700 1 -- 172.21.15.38:0/4207581114 shutdown_connections 2024-04-22T21:08:07.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.482+0000 7f20f9d5d700 1 --2- 172.21.15.38:0/4207581114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f40fff30 0x7f20f4102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:07.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.482+0000 7f20f9d5d700 1 --2- 172.21.15.38:0/4207581114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f4066f80 0x7f20f40ff9f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:07.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.482+0000 7f20f9d5d700 1 -- 172.21.15.38:0/4207581114 >> 172.21.15.38:0/4207581114 conn(0x7f20f40f7480 msgr2=0x7f20f40f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:08:07.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.482+0000 7f20f9d5d700 1 -- 172.21.15.38:0/4207581114 shutdown_connections 2024-04-22T21:08:07.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.482+0000 7f20f9d5d700 1 -- 172.21.15.38:0/4207581114 wait complete. 2024-04-22T21:08:07.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.483+0000 7f20f9d5d700 1 Processor -- start 2024-04-22T21:08:07.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.483+0000 7f20f9d5d700 1 -- start start 2024-04-22T21:08:07.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.483+0000 7f20f9d5d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f4066f80 0x7f20f40fd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:08:07.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.483+0000 7f20f9d5d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f40fff30 0x7f20f40fb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:08:07.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.483+0000 7f20f9d5d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f40fd840 con 0x7f20f40fff30 2024-04-22T21:08:07.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.483+0000 7f20f9d5d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f40fbe30 con 0x7f20f4066f80 2024-04-22T21:08:07.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.483+0000 7f20f2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f40fff30 0x7f20f40fb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:08:07.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.483+0000 7f20f2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f40fff30 0x7f20f40fb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41866/0 (socket says 172.21.15.38:41866) 2024-04-22T21:08:07.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.483+0000 7f20f2ffd700 1 -- 172.21.15.38:0/2755834721 learned_addr learned my addr 172.21.15.38:0/2755834721 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:08:07.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.483+0000 7f20f37fe700 1 --2- 172.21.15.38:0/2755834721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f4066f80 0x7f20f40fd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:08:07.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.484+0000 7f20f2ffd700 1 -- 172.21.15.38:0/2755834721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f4066f80 msgr2=0x7f20f40fd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:07.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.484+0000 7f20f2ffd700 1 --2- 172.21.15.38:0/2755834721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f4066f80 0x7f20f40fd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:07.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.484+0000 7f20f2ffd700 1 -- 172.21.15.38:0/2755834721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e4005040 con 0x7f20f40fff30 2024-04-22T21:08:07.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.484+0000 7f20f2ffd700 1 --2- 172.21.15.38:0/2755834721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f40fff30 0x7f20f40fb8f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f20dc00c8c0 tx=0x7f20dc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:08:07.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.484+0000 7f20ebfff700 1 -- 172.21.15.38:0/2755834721 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20dc00a760 con 0x7f20f40fff30 2024-04-22T21:08:07.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.484+0000 7f20f9d5d700 1 -- 172.21.15.38:0/2755834721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f40fc0e0 con 0x7f20f40fff30 2024-04-22T21:08:07.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.484+0000 7f20ebfff700 1 -- 172.21.15.38:0/2755834721 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f20dc00ad90 con 0x7f20f40fff30 2024-04-22T21:08:07.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.484+0000 7f20ebfff700 1 -- 172.21.15.38:0/2755834721 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20dc0113a0 con 0x7f20f40fff30 2024-04-22T21:08:07.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.484+0000 7f20f9d5d700 1 -- 172.21.15.38:0/2755834721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20f40fc6a0 con 0x7f20f40fff30 2024-04-22T21:08:07.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.485+0000 7f20f9d5d700 1 -- 172.21.15.38:0/2755834721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20f404c710 con 0x7f20f40fff30 2024-04-22T21:08:07.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.485+0000 7f20ebfff700 1 -- 172.21.15.38:0/2755834721 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f20dc012400 con 0x7f20f40fff30 2024-04-22T21:08:07.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.486+0000 7f20ebfff700 1 --2- 172.21.15.38:0/2755834721 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f20e00692a0 0x7f20e006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:08:07.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.486+0000 7f20ebfff700 1 -- 172.21.15.38:0/2755834721 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f20dc0872a0 con 0x7f20f40fff30 2024-04-22T21:08:07.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.486+0000 7f20f37fe700 1 --2- 172.21.15.38:0/2755834721 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f20e00692a0 0x7f20e006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:08:07.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.486+0000 7f20f37fe700 1 --2- 172.21.15.38:0/2755834721 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f20e00692a0 0x7f20e006b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f20e400b020 tx=0x7f20e4009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:08:07.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.491+0000 7f20ebfff700 1 -- 172.21.15.38:0/2755834721 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f20dc056f70 con 0x7f20f40fff30 2024-04-22T21:08:07.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:07 smithi098 ceph-mon[34053]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-04-22T21:08:07.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.688+0000 7f20f9d5d700 1 -- 172.21.15.38:0/2755834721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f20f41058e0 con 0x7f20f40fff30 2024-04-22T21:08:07.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.689+0000 7f20ebfff700 1 -- 172.21.15.38:0/2755834721 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f20dc056b00 con 0x7f20f40fff30 2024-04-22T21:08:07.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.691+0000 7f20f9d5d700 1 -- 172.21.15.38:0/2755834721 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f20e00692a0 msgr2=0x7f20e006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:07.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.691+0000 7f20f9d5d700 1 --2- 172.21.15.38:0/2755834721 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f20e00692a0 0x7f20e006b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f20e400b020 tx=0x7f20e4009e30 comp rx=0 tx=0).stop 2024-04-22T21:08:07.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.691+0000 7f20f9d5d700 1 -- 172.21.15.38:0/2755834721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f40fff30 msgr2=0x7f20f40fb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:07.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.691+0000 7f20f9d5d700 1 --2- 172.21.15.38:0/2755834721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f40fff30 0x7f20f40fb8f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f20dc00c8c0 tx=0x7f20dc0044c0 comp rx=0 tx=0).stop 2024-04-22T21:08:07.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.691+0000 7f20f9d5d700 1 -- 172.21.15.38:0/2755834721 shutdown_connections 2024-04-22T21:08:07.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.691+0000 7f20f9d5d700 1 --2- 172.21.15.38:0/2755834721 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f20e00692a0 0x7f20e006b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:07.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.691+0000 7f20f9d5d700 1 --2- 172.21.15.38:0/2755834721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f4066f80 0x7f20f40fd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:07.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.691+0000 7f20f9d5d700 1 --2- 172.21.15.38:0/2755834721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f40fff30 0x7f20f40fb8f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:07.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.691+0000 7f20f9d5d700 1 -- 172.21.15.38:0/2755834721 >> 172.21.15.38:0/2755834721 conn(0x7f20f40f7480 msgr2=0x7f20f40f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:08:07.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.691+0000 7f20f9d5d700 1 -- 172.21.15.38:0/2755834721 shutdown_connections 2024-04-22T21:08:07.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:07.691+0000 7f20f9d5d700 1 -- 172.21.15.38:0/2755834721 wait complete. 2024-04-22T21:08:08.194 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-22T21:08:08.460 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:08:08.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:08 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:08:08.583 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:08 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:08:09.493 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:09 smithi038 ceph-mon[26374]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-04-22T21:08:09.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:09 smithi098 ceph-mon[34053]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-04-22T21:08:10.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.020+0000 7fe657443700 1 -- 172.21.15.38:0/2010945879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6500fe440 msgr2=0x7fe6500fe850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:10.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.020+0000 7fe657443700 1 --2- 172.21.15.38:0/2010945879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6500fe440 0x7fe6500fe850 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe64c004940 tx=0x7fe64c009e30 comp rx=0 tx=0).stop 2024-04-22T21:08:10.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.020+0000 7fe657443700 1 -- 172.21.15.38:0/2010945879 shutdown_connections 2024-04-22T21:08:10.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.020+0000 7fe657443700 1 --2- 172.21.15.38:0/2010945879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6500fed90 0x7fe6500faed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:10.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.020+0000 7fe657443700 1 --2- 172.21.15.38:0/2010945879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6500fe440 0x7fe6500fe850 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:10.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.020+0000 7fe657443700 1 -- 172.21.15.38:0/2010945879 >> 172.21.15.38:0/2010945879 conn(0x7fe6500f6be0 msgr2=0x7fe6500f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:08:10.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.020+0000 7fe657443700 1 -- 172.21.15.38:0/2010945879 shutdown_connections 2024-04-22T21:08:10.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.020+0000 7fe657443700 1 -- 172.21.15.38:0/2010945879 wait complete. 2024-04-22T21:08:10.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.020+0000 7fe657443700 1 Processor -- start 2024-04-22T21:08:10.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.020+0000 7fe657443700 1 -- start start 2024-04-22T21:08:10.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.021+0000 7fe657443700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6500fe440 0x7fe65010d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:08:10.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.021+0000 7fe657443700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6500fed90 0x7fe65010de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:08:10.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.021+0000 7fe657443700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65010e460 con 0x7fe6500fed90 2024-04-22T21:08:10.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.021+0000 7fe657443700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65010e5d0 con 0x7fe6500fe440 2024-04-22T21:08:10.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.021+0000 7fe6549de700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6500fed90 0x7fe65010de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:08:10.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.021+0000 7fe6549de700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6500fed90 0x7fe65010de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41872/0 (socket says 172.21.15.38:41872) 2024-04-22T21:08:10.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.021+0000 7fe6549de700 1 -- 172.21.15.38:0/1136731921 learned_addr learned my addr 172.21.15.38:0/1136731921 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:08:10.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.021+0000 7fe6551df700 1 --2- 172.21.15.38:0/1136731921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6500fe440 0x7fe65010d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:08:10.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.021+0000 7fe6549de700 1 -- 172.21.15.38:0/1136731921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6500fe440 msgr2=0x7fe65010d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:10.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.021+0000 7fe6549de700 1 --2- 172.21.15.38:0/1136731921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6500fe440 0x7fe65010d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:10.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.021+0000 7fe6549de700 1 -- 172.21.15.38:0/1136731921 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c005040 con 0x7fe6500fed90 2024-04-22T21:08:10.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.021+0000 7fe6549de700 1 --2- 172.21.15.38:0/1136731921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6500fed90 0x7fe65010de60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe64000c8c0 tx=0x7fe6400044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:08:10.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.021+0000 7fe645ffb700 1 -- 172.21.15.38:0/1136731921 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe64000a980 con 0x7fe6500fed90 2024-04-22T21:08:10.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.022+0000 7fe645ffb700 1 -- 172.21.15.38:0/1136731921 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe64000ce60 con 0x7fe6500fed90 2024-04-22T21:08:10.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.022+0000 7fe645ffb700 1 -- 172.21.15.38:0/1136731921 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6400115e0 con 0x7fe6500fed90 2024-04-22T21:08:10.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.022+0000 7fe657443700 1 -- 172.21.15.38:0/1136731921 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe650113030 con 0x7fe6500fed90 2024-04-22T21:08:10.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.022+0000 7fe657443700 1 -- 172.21.15.38:0/1136731921 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6501135c0 con 0x7fe6500fed90 2024-04-22T21:08:10.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.023+0000 7fe657443700 1 -- 172.21.15.38:0/1136731921 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65004c710 con 0x7fe6500fed90 2024-04-22T21:08:10.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.023+0000 7fe645ffb700 1 -- 172.21.15.38:0/1136731921 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fe640012400 con 0x7fe6500fed90 2024-04-22T21:08:10.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.023+0000 7fe645ffb700 1 --2- 172.21.15.38:0/1136731921 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe63c0692a0 0x7fe63c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:08:10.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.024+0000 7fe645ffb700 1 -- 172.21.15.38:0/1136731921 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe6400872f0 con 0x7fe6500fed90 2024-04-22T21:08:10.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.024+0000 7fe6551df700 1 --2- 172.21.15.38:0/1136731921 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe63c0692a0 0x7fe63c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:08:10.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.024+0000 7fe6551df700 1 --2- 172.21.15.38:0/1136731921 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe63c0692a0 0x7fe63c06b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00b3b0 tx=0x7fe64c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:08:10.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.028+0000 7fe645ffb700 1 -- 172.21.15.38:0/1136731921 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe640057060 con 0x7fe6500fed90 2024-04-22T21:08:10.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.226+0000 7fe657443700 1 -- 172.21.15.38:0/1136731921 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fe6501058f0 con 0x7fe6500fed90 2024-04-22T21:08:10.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.227+0000 7fe645ffb700 1 -- 172.21.15.38:0/1136731921 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7fe6400126b0 con 0x7fe6500fed90 2024-04-22T21:08:10.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.229+0000 7fe657443700 1 -- 172.21.15.38:0/1136731921 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe63c0692a0 msgr2=0x7fe63c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:10.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.229+0000 7fe657443700 1 --2- 172.21.15.38:0/1136731921 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe63c0692a0 0x7fe63c06b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00b3b0 tx=0x7fe64c009e30 comp rx=0 tx=0).stop 2024-04-22T21:08:10.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.229+0000 7fe657443700 1 -- 172.21.15.38:0/1136731921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6500fed90 msgr2=0x7fe65010de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:10.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.229+0000 7fe657443700 1 --2- 172.21.15.38:0/1136731921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6500fed90 0x7fe65010de60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe64000c8c0 tx=0x7fe6400044c0 comp rx=0 tx=0).stop 2024-04-22T21:08:10.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.229+0000 7fe657443700 1 -- 172.21.15.38:0/1136731921 shutdown_connections 2024-04-22T21:08:10.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.229+0000 7fe657443700 1 --2- 172.21.15.38:0/1136731921 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fe63c0692a0 0x7fe63c06b750 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:10.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.229+0000 7fe657443700 1 --2- 172.21.15.38:0/1136731921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6500fe440 0x7fe65010d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:10.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.229+0000 7fe657443700 1 --2- 172.21.15.38:0/1136731921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6500fed90 0x7fe65010de60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:10.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.229+0000 7fe657443700 1 -- 172.21.15.38:0/1136731921 >> 172.21.15.38:0/1136731921 conn(0x7fe6500f6be0 msgr2=0x7fe6500f82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:08:10.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.229+0000 7fe657443700 1 -- 172.21.15.38:0/1136731921 shutdown_connections 2024-04-22T21:08:10.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:10.229+0000 7fe657443700 1 -- 172.21.15.38:0/1136731921 wait complete. 2024-04-22T21:08:10.796 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-04-22T21:08:11.063 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:08:11.405 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:11 smithi038 ceph-mon[26374]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-04-22T21:08:11.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:11 smithi098 ceph-mon[34053]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-04-22T21:08:12.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.501+0000 7f8f2c9f1700 1 -- 172.21.15.38:0/278568725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f280662f0 msgr2=0x7f8f280ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:12.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.501+0000 7f8f2c9f1700 1 --2- 172.21.15.38:0/278568725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f280662f0 0x7f8f280ffa30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8f18004940 tx=0x7f8f18009e30 comp rx=0 tx=0).stop 2024-04-22T21:08:12.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.501+0000 7f8f2c9f1700 1 -- 172.21.15.38:0/278568725 shutdown_connections 2024-04-22T21:08:12.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.501+0000 7f8f2c9f1700 1 --2- 172.21.15.38:0/278568725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f280fff70 0x7f8f28102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:12.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.501+0000 7f8f2c9f1700 1 --2- 172.21.15.38:0/278568725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f280662f0 0x7f8f280ffa30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:12.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.501+0000 7f8f2c9f1700 1 -- 172.21.15.38:0/278568725 >> 172.21.15.38:0/278568725 conn(0x7f8f280f74c0 msgr2=0x7f8f280f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:08:12.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.501+0000 7f8f2c9f1700 1 -- 172.21.15.38:0/278568725 shutdown_connections 2024-04-22T21:08:12.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.501+0000 7f8f2c9f1700 1 -- 172.21.15.38:0/278568725 wait complete. 2024-04-22T21:08:12.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.501+0000 7f8f2c9f1700 1 Processor -- start 2024-04-22T21:08:12.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.501+0000 7f8f2c9f1700 1 -- start start 2024-04-22T21:08:12.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.501+0000 7f8f2c9f1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f280662f0 0x7f8f280fd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:08:12.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.501+0000 7f8f2c9f1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f280fff70 0x7f8f280fb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:08:12.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.501+0000 7f8f2c9f1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f280fbf50 con 0x7f8f280fff70 2024-04-22T21:08:12.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.501+0000 7f8f2c9f1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f280fc090 con 0x7f8f280662f0 2024-04-22T21:08:12.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.502+0000 7f8f25d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f280fff70 0x7f8f280fb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:08:12.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.502+0000 7f8f25d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f280fff70 0x7f8f280fb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41894/0 (socket says 172.21.15.38:41894) 2024-04-22T21:08:12.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.502+0000 7f8f25d9b700 1 -- 172.21.15.38:0/1105767346 learned_addr learned my addr 172.21.15.38:0/1105767346 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:08:12.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.502+0000 7f8f2659c700 1 --2- 172.21.15.38:0/1105767346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f280662f0 0x7f8f280fd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:08:12.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.502+0000 7f8f25d9b700 1 -- 172.21.15.38:0/1105767346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f280662f0 msgr2=0x7f8f280fd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:12.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.502+0000 7f8f25d9b700 1 --2- 172.21.15.38:0/1105767346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f280662f0 0x7f8f280fd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:12.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.502+0000 7f8f25d9b700 1 -- 172.21.15.38:0/1105767346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f18005040 con 0x7f8f280fff70 2024-04-22T21:08:12.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.502+0000 7f8f2659c700 1 --2- 172.21.15.38:0/1105767346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f280662f0 0x7f8f280fd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T21:08:12.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.502+0000 7f8f25d9b700 1 --2- 172.21.15.38:0/1105767346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f280fff70 0x7f8f280fb930 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00b790 tx=0x7f8f1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:08:12.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.502+0000 7f8f16ffd700 1 -- 172.21.15.38:0/1105767346 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f1c00f770 con 0x7f8f280fff70 2024-04-22T21:08:12.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.502+0000 7f8f16ffd700 1 -- 172.21.15.38:0/1105767346 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f1c00fda0 con 0x7f8f280fff70 2024-04-22T21:08:12.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.502+0000 7f8f2c9f1700 1 -- 172.21.15.38:0/1105767346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f280fc370 con 0x7f8f280fff70 2024-04-22T21:08:12.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.502+0000 7f8f16ffd700 1 -- 172.21.15.38:0/1105767346 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f1c00d310 con 0x7f8f280fff70 2024-04-22T21:08:12.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.502+0000 7f8f2c9f1700 1 -- 172.21.15.38:0/1105767346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f28115ac0 con 0x7f8f280fff70 2024-04-22T21:08:12.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.504+0000 7f8f2c9f1700 1 -- 172.21.15.38:0/1105767346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f2805f5f0 con 0x7f8f280fff70 2024-04-22T21:08:12.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.504+0000 7f8f16ffd700 1 -- 172.21.15.38:0/1105767346 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f8f1c010420 con 0x7f8f280fff70 2024-04-22T21:08:12.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.505+0000 7f8f16ffd700 1 --2- 172.21.15.38:0/1105767346 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8f100692a0 0x7f8f1006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:08:12.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.505+0000 7f8f16ffd700 1 -- 172.21.15.38:0/1105767346 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8f1c087210 con 0x7f8f280fff70 2024-04-22T21:08:12.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.505+0000 7f8f2659c700 1 --2- 172.21.15.38:0/1105767346 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8f100692a0 0x7f8f1006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:08:12.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.505+0000 7f8f2659c700 1 --2- 172.21.15.38:0/1105767346 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8f100692a0 0x7f8f1006b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8f18004890 tx=0x7f8f18009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:08:12.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.509+0000 7f8f16ffd700 1 -- 172.21.15.38:0/1105767346 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8f1c0534f0 con 0x7f8f280fff70 2024-04-22T21:08:12.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:12.727+0000 7f8f2c9f1700 1 -- 172.21.15.38:0/1105767346 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f280f8fb0 con 0x7f8f100692a0 2024-04-22T21:08:13.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:13 smithi098 ceph-mon[34053]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-04-22T21:08:13.548 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:13 smithi038 ceph-mon[26374]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-04-22T21:08:14.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:14 smithi098 ceph-mon[34053]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:08:14.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:14 smithi038 ceph-mon[26374]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:08:15.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:15 smithi098 ceph-mon[34053]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-04-22T21:08:15.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:15 smithi038 ceph-mon[26374]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-04-22T21:08:17.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:17 smithi098 ceph-mon[34053]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-04-22T21:08:17.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:17 smithi038 ceph-mon[26374]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-04-22T21:08:19.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:19 smithi098 ceph-mon[34053]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-22T21:08:19.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:19 smithi038 ceph-mon[26374]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-22T21:08:21.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:21 smithi098 ceph-mon[34053]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:21.537 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:21 smithi038 ceph-mon[26374]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:23.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:23 smithi098 ceph-mon[34053]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:23.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:23 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:08:23.622 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:23 smithi038 ceph-mon[26374]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:23.622 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:23 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:08:25.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:25 smithi038 ceph-mon[26374]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:25.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:25 smithi098 ceph-mon[34053]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:27.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:27 smithi038 ceph-mon[26374]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:27.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:27 smithi098 ceph-mon[34053]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:29.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:29 smithi098 ceph-mon[34053]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:29.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:29 smithi038 ceph-mon[26374]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:31.535 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:31 smithi038 ceph-mon[26374]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:31.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:31 smithi098 ceph-mon[34053]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:33.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:33 smithi038 ceph-mon[26374]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:33.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:33 smithi098 ceph-mon[34053]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:35.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:35 smithi038 ceph-mon[26374]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:35.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:35 smithi098 ceph-mon[34053]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:37.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:37 smithi038 ceph-mon[26374]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:37.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:37 smithi098 ceph-mon[34053]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:08:38.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:08:38.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:38 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:08:39.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:39 smithi098 ceph-mon[34053]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-22T21:08:39.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:39 smithi038 ceph-mon[26374]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-22T21:08:41.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:41 smithi038 ceph-mon[26374]: pgmap v216: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-22T21:08:41.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:41 smithi098 ceph-mon[34053]: pgmap v216: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-22T21:08:43.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:43 smithi038 ceph-mon[26374]: pgmap v217: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-22T21:08:43.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:43 smithi098 ceph-mon[34053]: pgmap v217: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-22T21:08:45.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:45 smithi038 ceph-mon[26374]: pgmap v218: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-22T21:08:45.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:45 smithi098 ceph-mon[34053]: pgmap v218: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-22T21:08:47.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:47 smithi038 ceph-mon[26374]: pgmap v219: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-22T21:08:47.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:47 smithi098 ceph-mon[34053]: pgmap v219: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-22T21:08:49.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:49 smithi098 ceph-mon[34053]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-04-22T21:08:49.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:49 smithi038 ceph-mon[26374]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-04-22T21:08:51.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:51 smithi038 ceph-mon[26374]: pgmap v221: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-22T21:08:51.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:51 smithi098 ceph-mon[34053]: pgmap v221: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-22T21:08:53.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:53 smithi038 ceph-mon[26374]: pgmap v222: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-22T21:08:53.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:08:53.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:08:53.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:53 smithi098 ceph-mon[34053]: pgmap v222: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-22T21:08:53.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:08:53.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:08:54.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:54 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:08:54.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:54 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:08:54.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:54 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:08:54.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:54 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:08:54.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:54 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:08:54.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:54 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:08:55.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:55 smithi038 ceph-mon[26374]: pgmap v223: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-22T21:08:55.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:55 smithi098 ceph-mon[34053]: pgmap v223: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-22T21:08:57.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:57 smithi038 ceph-mon[26374]: pgmap v224: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-22T21:08:57.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:57 smithi098 ceph-mon[34053]: pgmap v224: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-22T21:08:59.270 INFO:teuthology.orchestra.run.smithi038.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-22T21:08:59.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:59.274+0000 7f8f16ffd700 1 -- 172.21.15.38:0/1105767346 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f8f280f8fb0 con 0x7f8f100692a0 2024-04-22T21:08:59.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:59.277+0000 7f8f2c9f1700 1 -- 172.21.15.38:0/1105767346 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8f100692a0 msgr2=0x7f8f1006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:59.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:59.277+0000 7f8f2c9f1700 1 --2- 172.21.15.38:0/1105767346 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8f100692a0 0x7f8f1006b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8f18004890 tx=0x7f8f18009e30 comp rx=0 tx=0).stop 2024-04-22T21:08:59.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:59.277+0000 7f8f2c9f1700 1 -- 172.21.15.38:0/1105767346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f280fff70 msgr2=0x7f8f280fb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:08:59.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:59.277+0000 7f8f2c9f1700 1 --2- 172.21.15.38:0/1105767346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f280fff70 0x7f8f280fb930 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00b790 tx=0x7f8f1c009250 comp rx=0 tx=0).stop 2024-04-22T21:08:59.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:59.277+0000 7f8f2c9f1700 1 -- 172.21.15.38:0/1105767346 shutdown_connections 2024-04-22T21:08:59.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:59.277+0000 7f8f2c9f1700 1 --2- 172.21.15.38:0/1105767346 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8f100692a0 0x7f8f1006b750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:59.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:59.277+0000 7f8f2c9f1700 1 --2- 172.21.15.38:0/1105767346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f280662f0 0x7f8f280fd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:59.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:59.277+0000 7f8f2c9f1700 1 --2- 172.21.15.38:0/1105767346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f280fff70 0x7f8f280fb930 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:08:59.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:59.277+0000 7f8f2c9f1700 1 -- 172.21.15.38:0/1105767346 >> 172.21.15.38:0/1105767346 conn(0x7f8f280f74c0 msgr2=0x7f8f280f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:08:59.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:59.277+0000 7f8f2c9f1700 1 -- 172.21.15.38:0/1105767346 shutdown_connections 2024-04-22T21:08:59.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:08:59.277+0000 7f8f2c9f1700 1 -- 172.21.15.38:0/1105767346 wait complete. 2024-04-22T21:08:59.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:59 smithi098 ceph-mon[34053]: pgmap v225: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-22T21:08:59.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:59 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:08:59.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:59 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:08:59.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:59 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:08:59.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:59 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:08:59.500 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:08:59 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:08:59.598 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:59 smithi038 ceph-mon[26374]: pgmap v225: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-22T21:08:59.598 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:59 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:08:59.598 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:59 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:08:59.598 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:59 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:08:59.599 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:59 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:08:59.599 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:08:59 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:08:59.992 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c5f5352-00eb-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-22T21:09:00.319 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/3c5f5352-00eb-11ef-bc93-c7b262605968/mon.smithi038/config 2024-04-22T21:09:00.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:00 smithi038 ceph-mon[26374]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-22T21:09:00.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:00 smithi098 ceph-mon[34053]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-22T21:09:01.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:01 smithi038 ceph-mon[26374]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-22T21:09:01.536 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:01 smithi038 ceph-mon[26374]: pgmap v226: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:01.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:01 smithi098 ceph-mon[34053]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-22T21:09:01.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:01 smithi098 ceph-mon[34053]: pgmap v226: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:02.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.369+0000 7f00f7ef5700 1 -- 172.21.15.38:0/525035543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f00fd2e0 msgr2=0x7f00f00fd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:02.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.369+0000 7f00f7ef5700 1 --2- 172.21.15.38:0/525035543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f00fd2e0 0x7f00f00fd6f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f00ec004940 tx=0x7f00ec009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:02.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.369+0000 7f00f7ef5700 1 -- 172.21.15.38:0/525035543 shutdown_connections 2024-04-22T21:09:02.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.369+0000 7f00f7ef5700 1 --2- 172.21.15.38:0/525035543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f00fdc30 0x7f00f00fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:02.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.369+0000 7f00f7ef5700 1 --2- 172.21.15.38:0/525035543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f00fd2e0 0x7f00f00fd6f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:02.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.369+0000 7f00f7ef5700 1 -- 172.21.15.38:0/525035543 >> 172.21.15.38:0/525035543 conn(0x7f00f00f6bd0 msgr2=0x7f00f00f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.369+0000 7f00f7ef5700 1 -- 172.21.15.38:0/525035543 shutdown_connections 2024-04-22T21:09:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.369+0000 7f00f7ef5700 1 -- 172.21.15.38:0/525035543 wait complete. 2024-04-22T21:09:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.370+0000 7f00f7ef5700 1 Processor -- start 2024-04-22T21:09:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.370+0000 7f00f7ef5700 1 -- start start 2024-04-22T21:09:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.370+0000 7f00f7ef5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f00fd2e0 0x7f00f010b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.370+0000 7f00f5c91700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f00fd2e0 0x7f00f010b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.370+0000 7f00f5c91700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f00fd2e0 0x7f00f010b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56872/0 (socket says 172.21.15.38:56872) 2024-04-22T21:09:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.370+0000 7f00f5c91700 1 -- 172.21.15.38:0/3697480894 learned_addr learned my addr 172.21.15.38:0/3697480894 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:09:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.370+0000 7f00f7ef5700 1 --2- 172.21.15.38:0/3697480894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f010bc80 0x7f00f0110cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.371+0000 7f00f7ef5700 1 -- 172.21.15.38:0/3697480894 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f010c140 con 0x7f00f00fd2e0 2024-04-22T21:09:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.371+0000 7f00f7ef5700 1 -- 172.21.15.38:0/3697480894 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f010c280 con 0x7f00f010bc80 2024-04-22T21:09:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.371+0000 7f00f5490700 1 --2- 172.21.15.38:0/3697480894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f010bc80 0x7f00f0110cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.371+0000 7f00f5c91700 1 -- 172.21.15.38:0/3697480894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f010bc80 msgr2=0x7f00f0110cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.371+0000 7f00f5c91700 1 --2- 172.21.15.38:0/3697480894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f010bc80 0x7f00f0110cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.371+0000 7f00f5c91700 1 -- 172.21.15.38:0/3697480894 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec005040 con 0x7f00f00fd2e0 2024-04-22T21:09:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.371+0000 7f00f5c91700 1 --2- 172.21.15.38:0/3697480894 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f00fd2e0 0x7f00f010b740 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f00ec000c00 tx=0x7f00ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.371+0000 7f00e67fc700 1 -- 172.21.15.38:0/3697480894 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00ec00e030 con 0x7f00f00fd2e0 2024-04-22T21:09:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.371+0000 7f00f7ef5700 1 -- 172.21.15.38:0/3697480894 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f01111f0 con 0x7f00f00fd2e0 2024-04-22T21:09:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.372+0000 7f00e67fc700 1 -- 172.21.15.38:0/3697480894 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f00ec013de0 con 0x7f00f00fd2e0 2024-04-22T21:09:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.372+0000 7f00e67fc700 1 -- 172.21.15.38:0/3697480894 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00ec01ce50 con 0x7f00f00fd2e0 2024-04-22T21:09:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.372+0000 7f00f7ef5700 1 -- 172.21.15.38:0/3697480894 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f01116e0 con 0x7f00f00fd2e0 2024-04-22T21:09:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.373+0000 7f00e67fc700 1 -- 172.21.15.38:0/3697480894 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f00ec01c400 con 0x7f00f00fd2e0 2024-04-22T21:09:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.373+0000 7f00f7ef5700 1 -- 172.21.15.38:0/3697480894 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f004c710 con 0x7f00f00fd2e0 2024-04-22T21:09:02.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.374+0000 7f00e67fc700 1 --2- 172.21.15.38:0/3697480894 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f00dc0692a0 0x7f00dc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:02.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.374+0000 7f00e67fc700 1 -- 172.21.15.38:0/3697480894 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f00ec088bd0 con 0x7f00f00fd2e0 2024-04-22T21:09:02.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.374+0000 7f00f5490700 1 --2- 172.21.15.38:0/3697480894 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f00dc0692a0 0x7f00dc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:02.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.375+0000 7f00f5490700 1 --2- 172.21.15.38:0/3697480894 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f00dc0692a0 0x7f00dc06b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f00e0009b80 tx=0x7f00e000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:02.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.381+0000 7f00e67fc700 1 -- 172.21.15.38:0/3697480894 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f00ec058940 con 0x7f00f00fd2e0 2024-04-22T21:09:02.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.649+0000 7f00f7ef5700 1 -- 172.21.15.38:0/3697480894 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f00f0111870 con 0x7f00dc0692a0 2024-04-22T21:09:02.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.651+0000 7f00e67fc700 1 -- 172.21.15.38:0/3697480894 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f00f0111870 con 0x7f00dc0692a0 2024-04-22T21:09:02.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.653+0000 7f00f7ef5700 1 -- 172.21.15.38:0/3697480894 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f00dc0692a0 msgr2=0x7f00dc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:02.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.653+0000 7f00f7ef5700 1 --2- 172.21.15.38:0/3697480894 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f00dc0692a0 0x7f00dc06b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f00e0009b80 tx=0x7f00e000b040 comp rx=0 tx=0).stop 2024-04-22T21:09:02.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.653+0000 7f00f7ef5700 1 -- 172.21.15.38:0/3697480894 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f00fd2e0 msgr2=0x7f00f010b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:02.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.653+0000 7f00f7ef5700 1 --2- 172.21.15.38:0/3697480894 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f00fd2e0 0x7f00f010b740 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f00ec000c00 tx=0x7f00ec009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:02.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.654+0000 7f00f7ef5700 1 -- 172.21.15.38:0/3697480894 shutdown_connections 2024-04-22T21:09:02.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.654+0000 7f00f7ef5700 1 --2- 172.21.15.38:0/3697480894 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f00dc0692a0 0x7f00dc06b750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:02.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.654+0000 7f00f7ef5700 1 --2- 172.21.15.38:0/3697480894 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f00fd2e0 0x7f00f010b740 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:02.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.654+0000 7f00f7ef5700 1 --2- 172.21.15.38:0/3697480894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00f010bc80 0x7f00f0110cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:02.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.654+0000 7f00f7ef5700 1 -- 172.21.15.38:0/3697480894 >> 172.21.15.38:0/3697480894 conn(0x7f00f00f6bd0 msgr2=0x7f00f00fb510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:02.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.654+0000 7f00f7ef5700 1 -- 172.21.15.38:0/3697480894 shutdown_connections 2024-04-22T21:09:02.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.654+0000 7f00f7ef5700 1 -- 172.21.15.38:0/3697480894 wait complete. 2024-04-22T21:09:02.664 INFO:teuthology.orchestra.run.smithi038.stdout:true 2024-04-22T21:09:02.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.792+0000 7ff745629700 1 -- 172.21.15.38:0/289836964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7400fcfb0 msgr2=0x7ff7400fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:02.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.792+0000 7ff745629700 1 --2- 172.21.15.38:0/289836964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7400fcfb0 0x7ff7400fd420 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff734004940 tx=0x7ff734009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:02.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.792+0000 7ff745629700 1 -- 172.21.15.38:0/289836964 shutdown_connections 2024-04-22T21:09:02.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.792+0000 7ff745629700 1 --2- 172.21.15.38:0/289836964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7400fcfb0 0x7ff7400fd420 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:02.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.792+0000 7ff745629700 1 --2- 172.21.15.38:0/289836964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7400fbe40 0x7ff7400fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:02.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.792+0000 7ff745629700 1 -- 172.21.15.38:0/289836964 >> 172.21.15.38:0/289836964 conn(0x7ff7400f7450 msgr2=0x7ff7400f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:02.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.792+0000 7ff745629700 1 -- 172.21.15.38:0/289836964 shutdown_connections 2024-04-22T21:09:02.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.792+0000 7ff745629700 1 -- 172.21.15.38:0/289836964 wait complete. 2024-04-22T21:09:02.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.794+0000 7ff745629700 1 Processor -- start 2024-04-22T21:09:02.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.794+0000 7ff745629700 1 -- start start 2024-04-22T21:09:02.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.794+0000 7ff745629700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7400fbe40 0x7ff74010b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:02.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.794+0000 7ff745629700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7400fcfb0 0x7ff74010bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:02.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.794+0000 7ff745629700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74010c320 con 0x7ff7400fcfb0 2024-04-22T21:09:02.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.794+0000 7ff745629700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff740111d80 con 0x7ff7400fbe40 2024-04-22T21:09:02.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.795+0000 7ff73effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7400fbe40 0x7ff74010b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:02.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.795+0000 7ff73effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7400fbe40 0x7ff74010b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.38:52008/0 (socket says 172.21.15.38:52008) 2024-04-22T21:09:02.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.795+0000 7ff73effd700 1 -- 172.21.15.38:0/3746029810 learned_addr learned my addr 172.21.15.38:0/3746029810 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:09:02.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.795+0000 7ff73e7fc700 1 --2- 172.21.15.38:0/3746029810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7400fcfb0 0x7ff74010bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:02.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.795+0000 7ff73effd700 1 -- 172.21.15.38:0/3746029810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7400fcfb0 msgr2=0x7ff74010bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:02.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.795+0000 7ff73effd700 1 --2- 172.21.15.38:0/3746029810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7400fcfb0 0x7ff74010bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:02.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.795+0000 7ff73effd700 1 -- 172.21.15.38:0/3746029810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff734005040 con 0x7ff7400fbe40 2024-04-22T21:09:02.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.795+0000 7ff73e7fc700 1 --2- 172.21.15.38:0/3746029810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7400fcfb0 0x7ff74010bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T21:09:02.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.796+0000 7ff73effd700 1 --2- 172.21.15.38:0/3746029810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7400fbe40 0x7ff74010b790 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff73000eb40 tx=0x7ff73000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:02.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.796+0000 7ff7277fe700 1 -- 172.21.15.38:0/3746029810 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff730007cc0 con 0x7ff7400fbe40 2024-04-22T21:09:02.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.796+0000 7ff7277fe700 1 -- 172.21.15.38:0/3746029810 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff73000f040 con 0x7ff7400fbe40 2024-04-22T21:09:02.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.796+0000 7ff7277fe700 1 -- 172.21.15.38:0/3746029810 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff730009620 con 0x7ff7400fbe40 2024-04-22T21:09:02.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.796+0000 7ff745629700 1 -- 172.21.15.38:0/3746029810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff740112000 con 0x7ff7400fbe40 2024-04-22T21:09:02.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.796+0000 7ff745629700 1 -- 172.21.15.38:0/3746029810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff740112540 con 0x7ff7400fbe40 2024-04-22T21:09:02.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.798+0000 7ff745629700 1 -- 172.21.15.38:0/3746029810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7401050b0 con 0x7ff7400fbe40 2024-04-22T21:09:02.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.800+0000 7ff7277fe700 1 -- 172.21.15.38:0/3746029810 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ff730009780 con 0x7ff7400fbe40 2024-04-22T21:09:02.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.801+0000 7ff7277fe700 1 --2- 172.21.15.38:0/3746029810 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff728069370 0x7ff72806b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:02.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.801+0000 7ff7277fe700 1 -- 172.21.15.38:0/3746029810 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff730086de0 con 0x7ff7400fbe40 2024-04-22T21:09:02.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.802+0000 7ff73e7fc700 1 --2- 172.21.15.38:0/3746029810 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff728069370 0x7ff72806b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:02.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.802+0000 7ff73e7fc700 1 --2- 172.21.15.38:0/3746029810 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff728069370 0x7ff72806b820 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff7340137d0 tx=0x7ff734009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:02.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:02.808+0000 7ff7277fe700 1 -- 172.21.15.38:0/3746029810 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff730053150 con 0x7ff7400fbe40 2024-04-22T21:09:03.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.085+0000 7ff745629700 1 -- 172.21.15.38:0/3746029810 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7400f9000 con 0x7ff728069370 2024-04-22T21:09:03.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.087+0000 7ff7277fe700 1 -- 172.21.15.38:0/3746029810 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7ff7400f9000 con 0x7ff728069370 2024-04-22T21:09:03.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.089+0000 7ff745629700 1 -- 172.21.15.38:0/3746029810 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff728069370 msgr2=0x7ff72806b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:03.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.090+0000 7ff745629700 1 --2- 172.21.15.38:0/3746029810 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff728069370 0x7ff72806b820 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff7340137d0 tx=0x7ff734009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:03.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.090+0000 7ff745629700 1 -- 172.21.15.38:0/3746029810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7400fbe40 msgr2=0x7ff74010b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:03.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.090+0000 7ff745629700 1 --2- 172.21.15.38:0/3746029810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7400fbe40 0x7ff74010b790 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff73000eb40 tx=0x7ff73000b040 comp rx=0 tx=0).stop 2024-04-22T21:09:03.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.090+0000 7ff745629700 1 -- 172.21.15.38:0/3746029810 shutdown_connections 2024-04-22T21:09:03.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.090+0000 7ff745629700 1 --2- 172.21.15.38:0/3746029810 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ff728069370 0x7ff72806b820 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:03.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.090+0000 7ff745629700 1 --2- 172.21.15.38:0/3746029810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7400fbe40 0x7ff74010b790 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:03.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.090+0000 7ff745629700 1 --2- 172.21.15.38:0/3746029810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7400fcfb0 0x7ff74010bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:03.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.090+0000 7ff745629700 1 -- 172.21.15.38:0/3746029810 >> 172.21.15.38:0/3746029810 conn(0x7ff7400f7450 msgr2=0x7ff7400f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:03.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.091+0000 7ff745629700 1 -- 172.21.15.38:0/3746029810 shutdown_connections 2024-04-22T21:09:03.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.091+0000 7ff745629700 1 -- 172.21.15.38:0/3746029810 wait complete. 2024-04-22T21:09:03.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.223+0000 7fa32d4c0700 1 -- 172.21.15.38:0/2163456429 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa318013b70 con 0x7fa3280fcc50 2024-04-22T21:09:03.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.224+0000 7fa330726700 1 -- 172.21.15.38:0/2163456429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3280fcc50 msgr2=0x7fa3280fd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:03.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.224+0000 7fa330726700 1 --2- 172.21.15.38:0/2163456429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3280fcc50 0x7fa3280fd0c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa318004940 tx=0x7fa318009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:03.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.224+0000 7fa330726700 1 -- 172.21.15.38:0/2163456429 shutdown_connections 2024-04-22T21:09:03.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.224+0000 7fa330726700 1 --2- 172.21.15.38:0/2163456429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3280fcc50 0x7fa3280fd0c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:03.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.224+0000 7fa330726700 1 --2- 172.21.15.38:0/2163456429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3280fc300 0x7fa3280fc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:03.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.224+0000 7fa330726700 1 -- 172.21.15.38:0/2163456429 >> 172.21.15.38:0/2163456429 conn(0x7fa3280f6bd0 msgr2=0x7fa3280f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:03.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.224+0000 7fa330726700 1 -- 172.21.15.38:0/2163456429 shutdown_connections 2024-04-22T21:09:03.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.224+0000 7fa330726700 1 -- 172.21.15.38:0/2163456429 wait complete. 2024-04-22T21:09:03.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.225+0000 7fa330726700 1 Processor -- start 2024-04-22T21:09:03.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.225+0000 7fa330726700 1 -- start start 2024-04-22T21:09:03.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.225+0000 7fa330726700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3280fc300 0x7fa32810c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:03.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.225+0000 7fa330726700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3280fcc50 0x7fa32810cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:03.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.225+0000 7fa330726700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32810d3e0 con 0x7fa3280fcc50 2024-04-22T21:09:03.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.225+0000 7fa330726700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32810dc30 con 0x7fa3280fc300 2024-04-22T21:09:03.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.225+0000 7fa32e4c2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3280fc300 0x7fa32810c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:03.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.225+0000 7fa32e4c2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3280fc300 0x7fa32810c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.38:52024/0 (socket says 172.21.15.38:52024) 2024-04-22T21:09:03.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.225+0000 7fa32e4c2700 1 -- 172.21.15.38:0/3292968193 learned_addr learned my addr 172.21.15.38:0/3292968193 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:09:03.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.225+0000 7fa32dcc1700 1 --2- 172.21.15.38:0/3292968193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3280fcc50 0x7fa32810cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:03.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.226+0000 7fa32e4c2700 1 -- 172.21.15.38:0/3292968193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3280fcc50 msgr2=0x7fa32810cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:03.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.226+0000 7fa32e4c2700 1 --2- 172.21.15.38:0/3292968193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3280fcc50 0x7fa32810cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:03.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.226+0000 7fa32e4c2700 1 -- 172.21.15.38:0/3292968193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa318005040 con 0x7fa3280fc300 2024-04-22T21:09:03.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.226+0000 7fa32e4c2700 1 --2- 172.21.15.38:0/3292968193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3280fc300 0x7fa32810c8a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa32400eb40 tx=0x7fa32400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:03.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.227+0000 7fa31effd700 1 -- 172.21.15.38:0/3292968193 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa324007cc0 con 0x7fa3280fc300 2024-04-22T21:09:03.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.227+0000 7fa330726700 1 -- 172.21.15.38:0/3292968193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32810de30 con 0x7fa3280fc300 2024-04-22T21:09:03.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.227+0000 7fa31effd700 1 -- 172.21.15.38:0/3292968193 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa32400f040 con 0x7fa3280fc300 2024-04-22T21:09:03.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.227+0000 7fa31effd700 1 -- 172.21.15.38:0/3292968193 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa32400ca00 con 0x7fa3280fc300 2024-04-22T21:09:03.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.227+0000 7fa330726700 1 -- 172.21.15.38:0/3292968193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa32810e2b0 con 0x7fa3280fc300 2024-04-22T21:09:03.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.229+0000 7fa31effd700 1 -- 172.21.15.38:0/3292968193 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fa32400cb60 con 0x7fa3280fc300 2024-04-22T21:09:03.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.230+0000 7fa330726700 1 -- 172.21.15.38:0/3292968193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa32805f5f0 con 0x7fa3280fc300 2024-04-22T21:09:03.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.230+0000 7fa31effd700 1 --2- 172.21.15.38:0/3292968193 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa314069440 0x7fa31406b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:03.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.230+0000 7fa31effd700 1 -- 172.21.15.38:0/3292968193 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa324086ca0 con 0x7fa3280fc300 2024-04-22T21:09:03.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.231+0000 7fa32dcc1700 1 --2- 172.21.15.38:0/3292968193 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa314069440 0x7fa31406b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:03.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.231+0000 7fa32dcc1700 1 --2- 172.21.15.38:0/3292968193 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa314069440 0x7fa31406b8f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa318009b30 tx=0x7fa318009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:03.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.239+0000 7fa31effd700 1 -- 172.21.15.38:0/3292968193 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa32404e250 con 0x7fa3280fc300 2024-04-22T21:09:03.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.627+0000 7fa330726700 1 -- 172.21.15.38:0/3292968193 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa32810e620 con 0x7fa314069440 2024-04-22T21:09:03.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:03 smithi038 ceph-mon[26374]: pgmap v227: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:03.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.637+0000 7fa31effd700 1 -- 172.21.15.38:0/3292968193 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fa32810e620 con 0x7fa314069440 2024-04-22T21:09:03.633 INFO:teuthology.orchestra.run.smithi038.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T21:09:03.633 INFO:teuthology.orchestra.run.smithi038.stdout:alertmanager.smithi038 smithi038 *:9093,9094 running (5m) 70s ago 6m 24.5M - 0.25.0 c8568f914cd2 28f5f3f389c1 2024-04-22T21:09:03.633 INFO:teuthology.orchestra.run.smithi038.stdout:crash.smithi038 smithi038 running (7m) 70s ago 7m 6665k - 17.2.7-884-g70d0cb4b cafd8ddac8fa ff0889f35e01 2024-04-22T21:09:03.634 INFO:teuthology.orchestra.run.smithi038.stdout:crash.smithi098 smithi098 running (6m) 75s ago 6m 6656k - 17.2.7-884-g70d0cb4b cafd8ddac8fa ced65820ce17 2024-04-22T21:09:03.634 INFO:teuthology.orchestra.run.smithi038.stdout:grafana.smithi038 smithi038 *:3000 running (5m) 70s ago 6m 83.2M - 9.4.7 954c08fa6188 849b80d16433 2024-04-22T21:09:03.634 INFO:teuthology.orchestra.run.smithi038.stdout:mds.cephfs.smithi038.hfegws smithi038 running (88s) 70s ago 88s 14.2M - 17.2.7-884-g70d0cb4b cafd8ddac8fa c9bc3b2faa16 2024-04-22T21:09:03.634 INFO:teuthology.orchestra.run.smithi038.stdout:mds.cephfs.smithi038.vmivbd smithi038 running (95s) 70s ago 95s 10.4M - 17.2.7-884-g70d0cb4b cafd8ddac8fa 5cc4c9d5a63c 2024-04-22T21:09:03.634 INFO:teuthology.orchestra.run.smithi038.stdout:mds.cephfs.smithi098.izcoza smithi098 running (86s) 75s ago 85s 14.6M - 17.2.7-884-g70d0cb4b cafd8ddac8fa 3ef691591a41 2024-04-22T21:09:03.634 INFO:teuthology.orchestra.run.smithi038.stdout:mds.cephfs.smithi098.mipeta smithi098 running (93s) 75s ago 92s 14.2M - 17.2.7-884-g70d0cb4b cafd8ddac8fa 539456953f6c 2024-04-22T21:09:03.634 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.smithi038.lxfgut smithi038 *:9283 running (8m) 70s ago 8m 481M - 17.2.7-884-g70d0cb4b cafd8ddac8fa ee36b4dede58 2024-04-22T21:09:03.634 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.smithi098.azqohf smithi098 *:8443,9283 running (6m) 75s ago 6m 435M - 17.2.7-884-g70d0cb4b cafd8ddac8fa c3fae1c785a2 2024-04-22T21:09:03.634 INFO:teuthology.orchestra.run.smithi038.stdout:mon.smithi038 smithi038 running (8m) 70s ago 8m 45.3M 2048M 17.2.7-884-g70d0cb4b cafd8ddac8fa 2137696f8742 2024-04-22T21:09:03.634 INFO:teuthology.orchestra.run.smithi038.stdout:mon.smithi098 smithi098 running (5m) 75s ago 5m 35.3M 2048M 17.2.7-884-g70d0cb4b cafd8ddac8fa e996b8c559ed 2024-04-22T21:09:03.634 INFO:teuthology.orchestra.run.smithi038.stdout:node-exporter.smithi038 smithi038 *:9100 running (7m) 70s ago 7m 15.7M - 1.5.0 0da6a335fe13 347a9e37455f 2024-04-22T21:09:03.635 INFO:teuthology.orchestra.run.smithi038.stdout:node-exporter.smithi098 smithi098 *:9100 running (6m) 75s ago 6m 17.2M - 1.5.0 0da6a335fe13 5274c36423cc 2024-04-22T21:09:03.635 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 smithi038 running (5m) 70s ago 5m 46.6M 4289M 17.2.7-884-g70d0cb4b cafd8ddac8fa 160468eff0dc 2024-04-22T21:09:03.635 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 smithi038 running (4m) 70s ago 4m 47.9M 4289M 17.2.7-884-g70d0cb4b cafd8ddac8fa d79042851a05 2024-04-22T21:09:03.635 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2 smithi038 running (4m) 70s ago 4m 46.7M 4289M 17.2.7-884-g70d0cb4b cafd8ddac8fa 8aaa1e69183c 2024-04-22T21:09:03.635 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3 smithi098 running (3m) 75s ago 3m 45.2M 5312M 17.2.7-884-g70d0cb4b cafd8ddac8fa a59b0f44b178 2024-04-22T21:09:03.635 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 smithi098 running (3m) 75s ago 3m 41.2M 5312M 17.2.7-884-g70d0cb4b cafd8ddac8fa 848d77af5440 2024-04-22T21:09:03.635 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 smithi098 running (2m) 75s ago 2m 43.6M 5312M 17.2.7-884-g70d0cb4b cafd8ddac8fa ef4d55a626c6 2024-04-22T21:09:03.635 INFO:teuthology.orchestra.run.smithi038.stdout:prometheus.smithi038 smithi038 *:9095 running (5m) 70s ago 6m 55.0M - 2.43.0 a07b618ecd1d 5cc3d358fbbd 2024-04-22T21:09:03.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.640+0000 7fa330726700 1 -- 172.21.15.38:0/3292968193 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa314069440 msgr2=0x7fa31406b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:03.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.640+0000 7fa330726700 1 --2- 172.21.15.38:0/3292968193 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa314069440 0x7fa31406b8f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa318009b30 tx=0x7fa318009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:03.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.640+0000 7fa330726700 1 -- 172.21.15.38:0/3292968193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3280fc300 msgr2=0x7fa32810c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.640+0000 7fa330726700 1 --2- 172.21.15.38:0/3292968193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3280fc300 0x7fa32810c8a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa32400eb40 tx=0x7fa32400b040 comp rx=0 tx=0).stop 2024-04-22T21:09:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.640+0000 7fa330726700 1 -- 172.21.15.38:0/3292968193 shutdown_connections 2024-04-22T21:09:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.640+0000 7fa330726700 1 --2- 172.21.15.38:0/3292968193 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa314069440 0x7fa31406b8f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.640+0000 7fa330726700 1 --2- 172.21.15.38:0/3292968193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3280fc300 0x7fa32810c8a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.640+0000 7fa330726700 1 --2- 172.21.15.38:0/3292968193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3280fcc50 0x7fa32810cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.640+0000 7fa330726700 1 -- 172.21.15.38:0/3292968193 >> 172.21.15.38:0/3292968193 conn(0x7fa3280f6bd0 msgr2=0x7fa3280f81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.641+0000 7fa330726700 1 -- 172.21.15.38:0/3292968193 shutdown_connections 2024-04-22T21:09:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.641+0000 7fa330726700 1 -- 172.21.15.38:0/3292968193 wait complete. 2024-04-22T21:09:03.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:03 smithi098 ceph-mon[34053]: pgmap v227: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:03.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.771+0000 7f900a042700 1 -- 172.21.15.38:0/2268279167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc096210 msgr2=0x7f8ffc096620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:03.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.771+0000 7f900a042700 1 --2- 172.21.15.38:0/2268279167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc096210 0x7f8ffc096620 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9000004940 tx=0x7f9000009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:03.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.772+0000 7f900a042700 1 -- 172.21.15.38:0/2268279167 shutdown_connections 2024-04-22T21:09:03.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.772+0000 7f900a042700 1 --2- 172.21.15.38:0/2268279167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ffc096b60 0x7f8ffc092ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:03.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.772+0000 7f900a042700 1 --2- 172.21.15.38:0/2268279167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc096210 0x7f8ffc096620 secure :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9000004940 tx=0x7f9000009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:03.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.772+0000 7f900a042700 1 -- 172.21.15.38:0/2268279167 >> 172.21.15.38:0/2268279167 conn(0x7f8ffc08e990 msgr2=0x7f8ffc090de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:03.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.772+0000 7f900a042700 1 -- 172.21.15.38:0/2268279167 shutdown_connections 2024-04-22T21:09:03.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.772+0000 7f900a042700 1 -- 172.21.15.38:0/2268279167 wait complete. 2024-04-22T21:09:03.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.774+0000 7f900a042700 1 Processor -- start 2024-04-22T21:09:03.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.774+0000 7f900a042700 1 -- start start 2024-04-22T21:09:03.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.775+0000 7f900a042700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc096b60 0x7f8ffc0a5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:03.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.775+0000 7f9009040700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc096b60 0x7f8ffc0a5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:03.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.775+0000 7f9009040700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc096b60 0x7f8ffc0a5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33976/0 (socket says 172.21.15.38:33976) 2024-04-22T21:09:03.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.776+0000 7f9009040700 1 -- 172.21.15.38:0/3168046801 learned_addr learned my addr 172.21.15.38:0/3168046801 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:09:03.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.776+0000 7f900a042700 1 --2- 172.21.15.38:0/3168046801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ffc0a5dd0 0x7f8ffc0aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:03.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.776+0000 7f900a042700 1 -- 172.21.15.38:0/3168046801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc0a6320 con 0x7f8ffc096b60 2024-04-22T21:09:03.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.776+0000 7f900a042700 1 -- 172.21.15.38:0/3168046801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc0a6460 con 0x7f8ffc0a5dd0 2024-04-22T21:09:03.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.776+0000 7f900883f700 1 --2- 172.21.15.38:0/3168046801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ffc0a5dd0 0x7f8ffc0aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:03.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.777+0000 7f9009040700 1 -- 172.21.15.38:0/3168046801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ffc0a5dd0 msgr2=0x7f8ffc0aae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:03.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.777+0000 7f9009040700 1 --2- 172.21.15.38:0/3168046801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ffc0a5dd0 0x7f8ffc0aae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:03.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.777+0000 7f9009040700 1 -- 172.21.15.38:0/3168046801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9000005040 con 0x7f8ffc096b60 2024-04-22T21:09:03.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.777+0000 7f900883f700 1 --2- 172.21.15.38:0/3168046801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ffc0a5dd0 0x7f8ffc0aae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T21:09:03.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.777+0000 7f9009040700 1 --2- 172.21.15.38:0/3168046801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc096b60 0x7f8ffc0a5890 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9000004940 tx=0x7f9000009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:03.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.778+0000 7f8ff9ffb700 1 -- 172.21.15.38:0/3168046801 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f900000e040 con 0x7f8ffc096b60 2024-04-22T21:09:03.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.778+0000 7f8ff9ffb700 1 -- 172.21.15.38:0/3168046801 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9000004290 con 0x7f8ffc096b60 2024-04-22T21:09:03.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.778+0000 7f8ff9ffb700 1 -- 172.21.15.38:0/3168046801 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f900001c3f0 con 0x7f8ffc096b60 2024-04-22T21:09:03.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.779+0000 7f900a042700 1 -- 172.21.15.38:0/3168046801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ffc0ab3a0 con 0x7f8ffc096b60 2024-04-22T21:09:03.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.779+0000 7f900a042700 1 -- 172.21.15.38:0/3168046801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ffc0ab8a0 con 0x7f8ffc096b60 2024-04-22T21:09:03.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.781+0000 7f8ff9ffb700 1 -- 172.21.15.38:0/3168046801 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f900000bbf0 con 0x7f8ffc096b60 2024-04-22T21:09:03.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.782+0000 7f8ff9ffb700 1 --2- 172.21.15.38:0/3168046801 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8ff0069360 0x7f8ff006b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:03.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.782+0000 7f8ff9ffb700 1 -- 172.21.15.38:0/3168046801 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9000089dd0 con 0x7f8ffc096b60 2024-04-22T21:09:03.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.782+0000 7f900883f700 1 --2- 172.21.15.38:0/3168046801 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8ff0069360 0x7f8ff006b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:03.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.783+0000 7f900a042700 1 -- 172.21.15.38:0/3168046801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ffc0050d0 con 0x7f8ffc096b60 2024-04-22T21:09:03.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.783+0000 7f900883f700 1 --2- 172.21.15.38:0/3168046801 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8ff0069360 0x7f8ff006b810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009fd0 tx=0x7f8ff40091a0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:03.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:03.791+0000 7f8ff9ffb700 1 -- 172.21.15.38:0/3168046801 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f900008d060 con 0x7f8ffc096b60 2024-04-22T21:09:04.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.314+0000 7f900a042700 1 -- 172.21.15.38:0/3168046801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8ffc0abb50 con 0x7f8ffc096b60 2024-04-22T21:09:04.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.316+0000 7f8ff9ffb700 1 -- 172.21.15.38:0/3168046801 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f9000057a20 con 0x7f8ffc096b60 2024-04-22T21:09:04.312 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T21:09:04.312 INFO:teuthology.orchestra.run.smithi038.stdout: "mon": { 2024-04-22T21:09:04.312 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph version 17.2.7-884-g70d0cb4b (70d0cb4b9d576828afb0a05dcf8732894d343ece) quincy (stable)": 2 2024-04-22T21:09:04.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T21:09:04.312 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": { 2024-04-22T21:09:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph version 17.2.7-884-g70d0cb4b (70d0cb4b9d576828afb0a05dcf8732894d343ece) quincy (stable)": 2 2024-04-22T21:09:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T21:09:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": { 2024-04-22T21:09:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph version 17.2.7-884-g70d0cb4b (70d0cb4b9d576828afb0a05dcf8732894d343ece) quincy (stable)": 6 2024-04-22T21:09:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T21:09:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: "mds": { 2024-04-22T21:09:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph version 17.2.7-884-g70d0cb4b (70d0cb4b9d576828afb0a05dcf8732894d343ece) quincy (stable)": 4 2024-04-22T21:09:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T21:09:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: "overall": { 2024-04-22T21:09:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph version 17.2.7-884-g70d0cb4b (70d0cb4b9d576828afb0a05dcf8732894d343ece) quincy (stable)": 14 2024-04-22T21:09:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T21:09:04.313 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T21:09:04.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.320+0000 7f900a042700 1 -- 172.21.15.38:0/3168046801 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8ff0069360 msgr2=0x7f8ff006b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:04.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.320+0000 7f900a042700 1 --2- 172.21.15.38:0/3168046801 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8ff0069360 0x7f8ff006b810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009fd0 tx=0x7f8ff40091a0 comp rx=0 tx=0).stop 2024-04-22T21:09:04.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.320+0000 7f900a042700 1 -- 172.21.15.38:0/3168046801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc096b60 msgr2=0x7f8ffc0a5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:04.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.320+0000 7f900a042700 1 --2- 172.21.15.38:0/3168046801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc096b60 0x7f8ffc0a5890 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9000004940 tx=0x7f9000009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:04.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.321+0000 7f900a042700 1 -- 172.21.15.38:0/3168046801 shutdown_connections 2024-04-22T21:09:04.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.321+0000 7f900a042700 1 --2- 172.21.15.38:0/3168046801 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f8ff0069360 0x7f8ff006b810 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:04.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.321+0000 7f900a042700 1 --2- 172.21.15.38:0/3168046801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc096b60 0x7f8ffc0a5890 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:04.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.321+0000 7f900a042700 1 --2- 172.21.15.38:0/3168046801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ffc0a5dd0 0x7f8ffc0aae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:04.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.321+0000 7f900a042700 1 -- 172.21.15.38:0/3168046801 >> 172.21.15.38:0/3168046801 conn(0x7f8ffc08e990 msgr2=0x7f8ffc090080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:04.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.321+0000 7f900a042700 1 -- 172.21.15.38:0/3168046801 shutdown_connections 2024-04-22T21:09:04.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.321+0000 7f900a042700 1 -- 172.21.15.38:0/3168046801 wait complete. 2024-04-22T21:09:04.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.436+0000 7f9f233c9700 1 -- 172.21.15.38:0/1064564192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c0fff30 msgr2=0x7f9f1c102350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:04.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.436+0000 7f9f233c9700 1 --2- 172.21.15.38:0/1064564192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c0fff30 0x7f9f1c102350 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c004940 tx=0x7f9f0c009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:04.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.436+0000 7f9f233c9700 1 -- 172.21.15.38:0/1064564192 shutdown_connections 2024-04-22T21:09:04.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.436+0000 7f9f233c9700 1 --2- 172.21.15.38:0/1064564192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c0fff30 0x7f9f1c102350 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:04.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.436+0000 7f9f233c9700 1 --2- 172.21.15.38:0/1064564192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f1c066f00 0x7f9f1c0ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:04.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.436+0000 7f9f233c9700 1 -- 172.21.15.38:0/1064564192 >> 172.21.15.38:0/1064564192 conn(0x7f9f1c0f7440 msgr2=0x7f9f1c0f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:04.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.436+0000 7f9f233c9700 1 -- 172.21.15.38:0/1064564192 shutdown_connections 2024-04-22T21:09:04.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.436+0000 7f9f233c9700 1 -- 172.21.15.38:0/1064564192 wait complete. 2024-04-22T21:09:04.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.437+0000 7f9f233c9700 1 Processor -- start 2024-04-22T21:09:04.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.437+0000 7f9f233c9700 1 -- start start 2024-04-22T21:09:04.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.437+0000 7f9f233c9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c066f00 0x7f9f1c10d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:04.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.437+0000 7f9f233c9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f1c0fff30 0x7f9f1c10de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:04.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.437+0000 7f9f233c9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c10e470 con 0x7f9f1c066f00 2024-04-22T21:09:04.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.437+0000 7f9f233c9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c10e5e0 con 0x7f9f1c0fff30 2024-04-22T21:09:04.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.437+0000 7f9f21165700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c066f00 0x7f9f1c10d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:04.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.438+0000 7f9f21165700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c066f00 0x7f9f1c10d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33990/0 (socket says 172.21.15.38:33990) 2024-04-22T21:09:04.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.438+0000 7f9f21165700 1 -- 172.21.15.38:0/2108392304 learned_addr learned my addr 172.21.15.38:0/2108392304 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:09:04.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.438+0000 7f9f20964700 1 --2- 172.21.15.38:0/2108392304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f1c0fff30 0x7f9f1c10de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:04.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.438+0000 7f9f21165700 1 -- 172.21.15.38:0/2108392304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f1c0fff30 msgr2=0x7f9f1c10de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:04.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.438+0000 7f9f21165700 1 --2- 172.21.15.38:0/2108392304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f1c0fff30 0x7f9f1c10de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:04.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.438+0000 7f9f21165700 1 -- 172.21.15.38:0/2108392304 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c005040 con 0x7f9f1c066f00 2024-04-22T21:09:04.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.438+0000 7f9f21165700 1 --2- 172.21.15.38:0/2108392304 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c066f00 0x7f9f1c10d930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800eb40 tx=0x7f9f1800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:04.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.438+0000 7f9f11ffb700 1 -- 172.21.15.38:0/2108392304 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f18007cc0 con 0x7f9f1c066f00 2024-04-22T21:09:04.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.438+0000 7f9f11ffb700 1 -- 172.21.15.38:0/2108392304 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f1800f040 con 0x7f9f1c066f00 2024-04-22T21:09:04.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.438+0000 7f9f11ffb700 1 -- 172.21.15.38:0/2108392304 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f180095e0 con 0x7f9f1c066f00 2024-04-22T21:09:04.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.438+0000 7f9f233c9700 1 -- 172.21.15.38:0/2108392304 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f1c0fb7d0 con 0x7f9f1c066f00 2024-04-22T21:09:04.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.439+0000 7f9f233c9700 1 -- 172.21.15.38:0/2108392304 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1c0fbd10 con 0x7f9f1c066f00 2024-04-22T21:09:04.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.440+0000 7f9f233c9700 1 -- 172.21.15.38:0/2108392304 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f1c04c710 con 0x7f9f1c066f00 2024-04-22T21:09:04.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.440+0000 7f9f11ffb700 1 -- 172.21.15.38:0/2108392304 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f9f18009740 con 0x7f9f1c066f00 2024-04-22T21:09:04.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.441+0000 7f9f11ffb700 1 --2- 172.21.15.38:0/2108392304 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f9f080692a0 0x7f9f0806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:04.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.441+0000 7f9f11ffb700 1 -- 172.21.15.38:0/2108392304 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9f18086bb0 con 0x7f9f1c066f00 2024-04-22T21:09:04.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.441+0000 7f9f20964700 1 --2- 172.21.15.38:0/2108392304 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f9f080692a0 0x7f9f0806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:04.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.442+0000 7f9f20964700 1 --2- 172.21.15.38:0/2108392304 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f9f080692a0 0x7f9f0806b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c013870 tx=0x7f9f0c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:04.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.450+0000 7f9f11ffb700 1 -- 172.21.15.38:0/2108392304 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f9f1808d080 con 0x7f9f1c066f00 2024-04-22T21:09:04.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:04 smithi038 ceph-mon[26374]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:09:04.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:04 smithi038 ceph-mon[26374]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:09:04.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:04 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:09:04.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:04 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:09:04.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:04 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:09:04.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:04 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/3168046801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:09:04.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.671+0000 7f9f233c9700 1 -- 172.21.15.38:0/2108392304 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f1c0f8fa0 con 0x7f9f080692a0 2024-04-22T21:09:04.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.672+0000 7f9f11ffb700 1 -- 172.21.15.38:0/2108392304 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f9f1c0f8fa0 con 0x7f9f080692a0 2024-04-22T21:09:04.668 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T21:09:04.668 INFO:teuthology.orchestra.run.smithi038.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:90c23f6ff09279e806b61a7a0a9b6863a8d239088a6c790da6364c3d6ab9ae8d", 2024-04-22T21:09:04.668 INFO:teuthology.orchestra.run.smithi038.stdout: "in_progress": true, 2024-04-22T21:09:04.668 INFO:teuthology.orchestra.run.smithi038.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T21:09:04.668 INFO:teuthology.orchestra.run.smithi038.stdout: "services_complete": [], 2024-04-22T21:09:04.668 INFO:teuthology.orchestra.run.smithi038.stdout: "progress": "0/2 daemons upgraded", 2024-04-22T21:09:04.669 INFO:teuthology.orchestra.run.smithi038.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:90c23f6ff09279e806b61a7a0a9b6863a8d239088a6c790da6364c3d6ab9ae8d image on host smithi098", 2024-04-22T21:09:04.669 INFO:teuthology.orchestra.run.smithi038.stdout: "is_paused": false 2024-04-22T21:09:04.669 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T21:09:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.675+0000 7f9f233c9700 1 -- 172.21.15.38:0/2108392304 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f9f080692a0 msgr2=0x7f9f0806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.675+0000 7f9f233c9700 1 --2- 172.21.15.38:0/2108392304 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f9f080692a0 0x7f9f0806b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c013870 tx=0x7f9f0c009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.675+0000 7f9f233c9700 1 -- 172.21.15.38:0/2108392304 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c066f00 msgr2=0x7f9f1c10d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.675+0000 7f9f233c9700 1 --2- 172.21.15.38:0/2108392304 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c066f00 0x7f9f1c10d930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800eb40 tx=0x7f9f1800b040 comp rx=0 tx=0).stop 2024-04-22T21:09:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.675+0000 7f9f233c9700 1 -- 172.21.15.38:0/2108392304 shutdown_connections 2024-04-22T21:09:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.675+0000 7f9f233c9700 1 --2- 172.21.15.38:0/2108392304 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f9f080692a0 0x7f9f0806b750 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.675+0000 7f9f233c9700 1 --2- 172.21.15.38:0/2108392304 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c066f00 0x7f9f1c10d930 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.675+0000 7f9f233c9700 1 --2- 172.21.15.38:0/2108392304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f1c0fff30 0x7f9f1c10de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.675+0000 7f9f233c9700 1 -- 172.21.15.38:0/2108392304 >> 172.21.15.38:0/2108392304 conn(0x7f9f1c0f7440 msgr2=0x7f9f1c0f7880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.675+0000 7f9f233c9700 1 -- 172.21.15.38:0/2108392304 shutdown_connections 2024-04-22T21:09:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:04.675+0000 7f9f233c9700 1 -- 172.21.15.38:0/2108392304 wait complete. 2024-04-22T21:09:04.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:04 smithi098 ceph-mon[34053]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:09:04.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:04 smithi098 ceph-mon[34053]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:09:04.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:09:04.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:09:04.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:04 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:09:04.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:04 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/3168046801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:09:05.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:05 smithi038 ceph-mon[26374]: from='client.24351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:09:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:05 smithi038 ceph-mon[26374]: Upgrade: Target is version 19.0.0-3198-g56e81df3 (unknown) 2024-04-22T21:09:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:05 smithi038 ceph-mon[26374]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:90c23f6ff09279e806b61a7a0a9b6863a8d239088a6c790da6364c3d6ab9ae8d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:90c23f6ff09279e806b61a7a0a9b6863a8d239088a6c790da6364c3d6ab9ae8d'] 2024-04-22T21:09:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:05 smithi038 ceph-mon[26374]: Upgrade: Need to upgrade myself (mgr.smithi038.lxfgut) 2024-04-22T21:09:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:05 smithi038 ceph-mon[26374]: pgmap v228: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:05.625 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:05 smithi038 ceph-mon[26374]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:90c23f6ff09279e806b61a7a0a9b6863a8d239088a6c790da6364c3d6ab9ae8d on smithi098 2024-04-22T21:09:05.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:05 smithi098 ceph-mon[34053]: from='client.24351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:09:05.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:05 smithi098 ceph-mon[34053]: Upgrade: Target is version 19.0.0-3198-g56e81df3 (unknown) 2024-04-22T21:09:05.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:05 smithi098 ceph-mon[34053]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:90c23f6ff09279e806b61a7a0a9b6863a8d239088a6c790da6364c3d6ab9ae8d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:90c23f6ff09279e806b61a7a0a9b6863a8d239088a6c790da6364c3d6ab9ae8d'] 2024-04-22T21:09:05.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:05 smithi098 ceph-mon[34053]: Upgrade: Need to upgrade myself (mgr.smithi038.lxfgut) 2024-04-22T21:09:05.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:05 smithi098 ceph-mon[34053]: pgmap v228: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:05.750 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:05 smithi098 ceph-mon[34053]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:90c23f6ff09279e806b61a7a0a9b6863a8d239088a6c790da6364c3d6ab9ae8d on smithi098 2024-04-22T21:09:06.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:06 smithi038 ceph-mon[26374]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:09:06.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:06 smithi098 ceph-mon[34053]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:09:07.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:07 smithi038 ceph-mon[26374]: pgmap v229: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:07.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:07 smithi098 ceph-mon[34053]: pgmap v229: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:08.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:08 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:09:08.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:08 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:09:09.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:09 smithi098 ceph-mon[34053]: pgmap v230: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:09.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:09 smithi038 ceph-mon[26374]: pgmap v230: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:11.676 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:11 smithi038 ceph-mon[26374]: pgmap v231: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:11.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:11 smithi098 ceph-mon[34053]: pgmap v231: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:13.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:13 smithi098 ceph-mon[34053]: pgmap v232: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:13.873 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:13 smithi038 ceph-mon[26374]: pgmap v232: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:15.444 INFO:tasks.workunit.client.1.smithi098.stderr:Updating files: 94% (11748/12484) Updating files: 95% (11860/12484) Updating files: 96% (11985/12484) Updating files: 97% (12110/12484) Updating files: 98% (12235/12484) Updating files: 99% (12360/12484) Updating files: 100% (12484/12484) Updating files: 100% (12484/12484), done. 2024-04-22T21:09:15.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:15 smithi098 ceph-mon[34053]: pgmap v233: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:15.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:15 smithi038 ceph-mon[26374]: pgmap v233: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:15.897 INFO:tasks.workunit.client.1.smithi098.stderr:Note: switching to '8254cb5a247caa8e3d35e534a1fefe2362a0a4b8'. 2024-04-22T21:09:15.897 INFO:tasks.workunit.client.1.smithi098.stderr: 2024-04-22T21:09:15.898 INFO:tasks.workunit.client.1.smithi098.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-22T21:09:15.898 INFO:tasks.workunit.client.1.smithi098.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-22T21:09:15.898 INFO:tasks.workunit.client.1.smithi098.stderr:state without impacting any branches by switching back to a branch. 2024-04-22T21:09:15.898 INFO:tasks.workunit.client.1.smithi098.stderr: 2024-04-22T21:09:15.898 INFO:tasks.workunit.client.1.smithi098.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-22T21:09:15.898 INFO:tasks.workunit.client.1.smithi098.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-22T21:09:15.898 INFO:tasks.workunit.client.1.smithi098.stderr: 2024-04-22T21:09:15.898 INFO:tasks.workunit.client.1.smithi098.stderr: git switch -c 2024-04-22T21:09:15.898 INFO:tasks.workunit.client.1.smithi098.stderr: 2024-04-22T21:09:15.898 INFO:tasks.workunit.client.1.smithi098.stderr:Or undo this operation with: 2024-04-22T21:09:15.898 INFO:tasks.workunit.client.1.smithi098.stderr: 2024-04-22T21:09:15.898 INFO:tasks.workunit.client.1.smithi098.stderr: git switch - 2024-04-22T21:09:15.898 INFO:tasks.workunit.client.1.smithi098.stderr: 2024-04-22T21:09:15.898 INFO:tasks.workunit.client.1.smithi098.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-22T21:09:15.899 INFO:tasks.workunit.client.1.smithi098.stderr: 2024-04-22T21:09:15.899 INFO:tasks.workunit.client.1.smithi098.stderr:HEAD is now at 8254cb5a247 Merge PR #54067 into main 2024-04-22T21:09:15.900 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-22T21:09:15.967 INFO:tasks.workunit.client.1.smithi098.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-22T21:09:15.969 INFO:tasks.workunit.client.1.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-22T21:09:15.969 INFO:tasks.workunit.client.1.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-22T21:09:16.631 INFO:tasks.workunit.client.1.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-22T21:09:16.689 INFO:tasks.workunit.client.1.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-22T21:09:16.729 INFO:tasks.workunit.client.1.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-22T21:09:16.730 INFO:tasks.workunit.client.1.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-22T21:09:16.731 INFO:tasks.workunit.client.1.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-22T21:09:16.778 INFO:tasks.workunit.client.1.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-22T21:09:16.781 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-22T21:09:16.781 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-22T21:09:16.836 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-04-22T21:09:16.837 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-22T21:09:16.838 DEBUG:teuthology.orchestra.run.smithi098:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-22T21:09:16.897 INFO:tasks.workunit.client.1.smithi098.stderr:+ mkdir -p fsstress 2024-04-22T21:09:16.899 INFO:tasks.workunit.client.1.smithi098.stderr:+ pushd fsstress 2024-04-22T21:09:16.899 INFO:tasks.workunit.client.1.smithi098.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-22T21:09:16.899 INFO:tasks.workunit.client.1.smithi098.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-22T21:09:16.989 INFO:tasks.workunit.client.0.smithi038.stderr:Updating files: 97% (12154/12484) Updating files: 98% (12235/12484) Updating files: 99% (12360/12484) Updating files: 100% (12484/12484) Updating files: 100% (12484/12484), done. 2024-04-22T21:09:17.039 INFO:tasks.workunit.client.0.smithi038.stderr:Note: switching to '8254cb5a247caa8e3d35e534a1fefe2362a0a4b8'. 2024-04-22T21:09:17.039 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T21:09:17.039 INFO:tasks.workunit.client.0.smithi038.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-22T21:09:17.039 INFO:tasks.workunit.client.0.smithi038.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-22T21:09:17.039 INFO:tasks.workunit.client.0.smithi038.stderr:state without impacting any branches by switching back to a branch. 2024-04-22T21:09:17.040 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T21:09:17.040 INFO:tasks.workunit.client.0.smithi038.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-22T21:09:17.040 INFO:tasks.workunit.client.0.smithi038.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-22T21:09:17.040 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T21:09:17.040 INFO:tasks.workunit.client.0.smithi038.stderr: git switch -c 2024-04-22T21:09:17.040 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T21:09:17.040 INFO:tasks.workunit.client.0.smithi038.stderr:Or undo this operation with: 2024-04-22T21:09:17.040 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T21:09:17.040 INFO:tasks.workunit.client.0.smithi038.stderr: git switch - 2024-04-22T21:09:17.040 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T21:09:17.040 INFO:tasks.workunit.client.0.smithi038.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-22T21:09:17.040 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T21:09:17.041 INFO:tasks.workunit.client.0.smithi038.stderr:HEAD is now at 8254cb5a247 Merge PR #54067 into main 2024-04-22T21:09:17.042 DEBUG:teuthology.orchestra.run.smithi038:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-22T21:09:17.073 INFO:tasks.workunit.client.0.smithi038.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-22T21:09:17.075 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T21:09:17.076 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-22T21:09:17.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:17 smithi098 ceph-mon[34053]: pgmap v234: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:17.750 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-22T21:09:17.805 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-22T21:09:17.847 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T21:09:17.848 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T21:09:17.849 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-22T21:09:17.873 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:17 smithi038 ceph-mon[26374]: pgmap v234: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:17.888 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T21:09:17.891 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T21:09:17.892 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-22T21:09:17.948 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-04-22T21:09:17.949 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-22T21:09:17.949 DEBUG:teuthology.orchestra.run.smithi038:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-04-22T21:09:18.009 INFO:tasks.workunit.client.0.smithi038.stderr:+ mkdir -p fsstress 2024-04-22T21:09:18.011 INFO:tasks.workunit.client.0.smithi038.stderr:+ pushd fsstress 2024-04-22T21:09:18.011 INFO:tasks.workunit.client.0.smithi038.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-22T21:09:18.012 INFO:tasks.workunit.client.0.smithi038.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-22T21:09:19.448 INFO:tasks.workunit.client.1.smithi098.stderr:+ tar xzf ltp-full.tgz 2024-04-22T21:09:19.483 INFO:tasks.workunit.client.0.smithi038.stderr:+ tar xzf ltp-full.tgz 2024-04-22T21:09:19.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:19 smithi098 ceph-mon[34053]: pgmap v235: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:19.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:19 smithi038 ceph-mon[26374]: pgmap v235: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-22T21:09:20.749 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:20 smithi098 ceph-mon[34053]: pgmap v236: 65 pgs: 65 active+clean; 620 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s wr, 0 op/s 2024-04-22T21:09:20.873 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:20 smithi038 ceph-mon[26374]: pgmap v236: 65 pgs: 65 active+clean; 620 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s wr, 0 op/s 2024-04-22T21:09:23.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:23 smithi098 ceph-mon[34053]: pgmap v237: 65 pgs: 65 active+clean; 620 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s wr, 0 op/s 2024-04-22T21:09:23.501 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:23 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:09:23.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:23 smithi038 ceph-mon[26374]: pgmap v237: 65 pgs: 65 active+clean; 620 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s wr, 0 op/s 2024-04-22T21:09:23.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:23 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:09:25.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:25 smithi098 ceph-mon[34053]: pgmap v238: 65 pgs: 65 active+clean; 3.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 230 KiB/s wr, 0 op/s 2024-04-22T21:09:25.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:25 smithi038 ceph-mon[26374]: pgmap v238: 65 pgs: 65 active+clean; 3.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 230 KiB/s wr, 0 op/s 2024-04-22T21:09:27.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:27 smithi098 ceph-mon[34053]: pgmap v239: 65 pgs: 65 active+clean; 18 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s wr, 104 op/s 2024-04-22T21:09:27.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:27 smithi038 ceph-mon[26374]: pgmap v239: 65 pgs: 65 active+clean; 18 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s wr, 104 op/s 2024-04-22T21:09:29.498 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:29 smithi098 ceph-mon[34053]: pgmap v240: 65 pgs: 65 active+clean; 48 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.1 MiB/s wr, 265 op/s 2024-04-22T21:09:29.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:29 smithi038 ceph-mon[26374]: pgmap v240: 65 pgs: 65 active+clean; 48 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.1 MiB/s wr, 265 op/s 2024-04-22T21:09:31.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:31 smithi098 ceph-mon[34053]: pgmap v241: 65 pgs: 65 active+clean; 52 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.5 MiB/s wr, 274 op/s 2024-04-22T21:09:31.534 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:31 smithi038 ceph-mon[26374]: pgmap v241: 65 pgs: 65 active+clean; 52 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.5 MiB/s wr, 274 op/s 2024-04-22T21:09:33.498 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:33 smithi098 ceph-mon[34053]: pgmap v242: 65 pgs: 65 active+clean; 56 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.8 MiB/s wr, 274 op/s 2024-04-22T21:09:33.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:33 smithi038 ceph-mon[26374]: pgmap v242: 65 pgs: 65 active+clean; 56 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.8 MiB/s wr, 274 op/s 2024-04-22T21:09:34.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.787+0000 7ffb5d6d6700 1 -- 172.21.15.38:0/4007604502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb580ff000 msgr2=0x7ffb580ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:34.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.787+0000 7ffb5d6d6700 1 --2- 172.21.15.38:0/4007604502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb580ff000 0x7ffb580ff410 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ffb50009ce0 tx=0x7ffb50004d10 comp rx=0 tx=0).stop 2024-04-22T21:09:34.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.788+0000 7ffb5d6d6700 1 -- 172.21.15.38:0/4007604502 shutdown_connections 2024-04-22T21:09:34.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.788+0000 7ffb5d6d6700 1 --2- 172.21.15.38:0/4007604502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb581002b0 0x7ffb58100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:34.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.788+0000 7ffb5d6d6700 1 --2- 172.21.15.38:0/4007604502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb580ff000 0x7ffb580ff410 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:34.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.788+0000 7ffb5d6d6700 1 -- 172.21.15.38:0/4007604502 >> 172.21.15.38:0/4007604502 conn(0x7ffb580fa950 msgr2=0x7ffb580fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:34.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.788+0000 7ffb5d6d6700 1 -- 172.21.15.38:0/4007604502 shutdown_connections 2024-04-22T21:09:34.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.788+0000 7ffb5d6d6700 1 -- 172.21.15.38:0/4007604502 wait complete. 2024-04-22T21:09:34.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.788+0000 7ffb5d6d6700 1 Processor -- start 2024-04-22T21:09:34.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.788+0000 7ffb5d6d6700 1 -- start start 2024-04-22T21:09:34.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.789+0000 7ffb5d6d6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb580ff000 0x7ffb5810fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:34.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.789+0000 7ffb5d6d6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb581002b0 0x7ffb581100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:34.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.789+0000 7ffb5d6d6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb581106b0 con 0x7ffb581002b0 2024-04-22T21:09:34.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.789+0000 7ffb5d6d6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb58110820 con 0x7ffb580ff000 2024-04-22T21:09:34.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.789+0000 7ffb567fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb581002b0 0x7ffb581100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:34.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.789+0000 7ffb567fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb581002b0 0x7ffb581100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58342/0 (socket says 172.21.15.38:58342) 2024-04-22T21:09:34.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.789+0000 7ffb56ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb580ff000 0x7ffb5810fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:34.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.789+0000 7ffb567fc700 1 -- 172.21.15.38:0/3644622745 learned_addr learned my addr 172.21.15.38:0/3644622745 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:09:34.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.789+0000 7ffb56ffd700 1 -- 172.21.15.38:0/3644622745 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb581002b0 msgr2=0x7ffb581100b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:34.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.789+0000 7ffb56ffd700 1 --2- 172.21.15.38:0/3644622745 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb581002b0 0x7ffb581100b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:34.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.789+0000 7ffb56ffd700 1 -- 172.21.15.38:0/3644622745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb50009940 con 0x7ffb580ff000 2024-04-22T21:09:34.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.789+0000 7ffb567fc700 1 --2- 172.21.15.38:0/3644622745 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb581002b0 0x7ffb581100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T21:09:34.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.789+0000 7ffb56ffd700 1 --2- 172.21.15.38:0/3644622745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb580ff000 0x7ffb5810fb70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffb50009cb0 tx=0x7ffb50004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:34.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.790+0000 7ffb4f7fe700 1 -- 172.21.15.38:0/3644622745 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb5000e040 con 0x7ffb580ff000 2024-04-22T21:09:34.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.790+0000 7ffb5d6d6700 1 -- 172.21.15.38:0/3644622745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb58115220 con 0x7ffb580ff000 2024-04-22T21:09:34.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.790+0000 7ffb4f7fe700 1 -- 172.21.15.38:0/3644622745 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb5001cb20 con 0x7ffb580ff000 2024-04-22T21:09:34.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.790+0000 7ffb4f7fe700 1 -- 172.21.15.38:0/3644622745 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb50012b10 con 0x7ffb580ff000 2024-04-22T21:09:34.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.790+0000 7ffb5d6d6700 1 -- 172.21.15.38:0/3644622745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb581157e0 con 0x7ffb580ff000 2024-04-22T21:09:34.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.791+0000 7ffb5d6d6700 1 -- 172.21.15.38:0/3644622745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5804c710 con 0x7ffb580ff000 2024-04-22T21:09:34.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.791+0000 7ffb4f7fe700 1 -- 172.21.15.38:0/3644622745 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ffb5001cc90 con 0x7ffb580ff000 2024-04-22T21:09:34.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.791+0000 7ffb4f7fe700 1 --2- 172.21.15.38:0/3644622745 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ffb440692a0 0x7ffb4406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:34.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.792+0000 7ffb4f7fe700 1 -- 172.21.15.38:0/3644622745 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ffb500894b0 con 0x7ffb580ff000 2024-04-22T21:09:34.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.792+0000 7ffb567fc700 1 --2- 172.21.15.38:0/3644622745 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ffb440692a0 0x7ffb4406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:34.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.792+0000 7ffb567fc700 1 --2- 172.21.15.38:0/3644622745 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ffb440692a0 0x7ffb4406b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ffb400041d0 tx=0x7ffb4000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:34.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:34.797+0000 7ffb4f7fe700 1 -- 172.21.15.38:0/3644622745 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ffb50059140 con 0x7ffb580ff000 2024-04-22T21:09:35.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.010+0000 7ffb5d6d6700 1 -- 172.21.15.38:0/3644622745 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb58104670 con 0x7ffb440692a0 2024-04-22T21:09:35.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.011+0000 7ffb4f7fe700 1 -- 172.21.15.38:0/3644622745 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7ffb58104670 con 0x7ffb440692a0 2024-04-22T21:09:35.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.013+0000 7ffb5d6d6700 1 -- 172.21.15.38:0/3644622745 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ffb440692a0 msgr2=0x7ffb4406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:35.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.013+0000 7ffb5d6d6700 1 --2- 172.21.15.38:0/3644622745 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ffb440692a0 0x7ffb4406b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ffb400041d0 tx=0x7ffb4000b040 comp rx=0 tx=0).stop 2024-04-22T21:09:35.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.013+0000 7ffb5d6d6700 1 -- 172.21.15.38:0/3644622745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb580ff000 msgr2=0x7ffb5810fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:35.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.013+0000 7ffb5d6d6700 1 --2- 172.21.15.38:0/3644622745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb580ff000 0x7ffb5810fb70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffb50009cb0 tx=0x7ffb50004d10 comp rx=0 tx=0).stop 2024-04-22T21:09:35.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.013+0000 7ffb5d6d6700 1 -- 172.21.15.38:0/3644622745 shutdown_connections 2024-04-22T21:09:35.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.013+0000 7ffb5d6d6700 1 --2- 172.21.15.38:0/3644622745 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7ffb440692a0 0x7ffb4406b750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.013+0000 7ffb5d6d6700 1 --2- 172.21.15.38:0/3644622745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb580ff000 0x7ffb5810fb70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.013+0000 7ffb5d6d6700 1 --2- 172.21.15.38:0/3644622745 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb581002b0 0x7ffb581100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.013+0000 7ffb5d6d6700 1 -- 172.21.15.38:0/3644622745 >> 172.21.15.38:0/3644622745 conn(0x7ffb580fa950 msgr2=0x7ffb580fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:35.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.013+0000 7ffb5d6d6700 1 -- 172.21.15.38:0/3644622745 shutdown_connections 2024-04-22T21:09:35.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.013+0000 7ffb5d6d6700 1 -- 172.21.15.38:0/3644622745 wait complete. 2024-04-22T21:09:35.018 INFO:teuthology.orchestra.run.smithi038.stdout:true 2024-04-22T21:09:35.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.119+0000 7fa26a910700 1 -- 172.21.15.38:0/2347883587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa264066b80 msgr2=0x7fa2640ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:35.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.119+0000 7fa26a910700 1 --2- 172.21.15.38:0/2347883587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa264066b80 0x7fa2640ffa70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa254004940 tx=0x7fa254009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:35.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.119+0000 7fa26a910700 1 -- 172.21.15.38:0/2347883587 shutdown_connections 2024-04-22T21:09:35.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.119+0000 7fa26a910700 1 --2- 172.21.15.38:0/2347883587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2640fffb0 0x7fa2641023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.119+0000 7fa26a910700 1 --2- 172.21.15.38:0/2347883587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa264066b80 0x7fa2640ffa70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.119+0000 7fa26a910700 1 -- 172.21.15.38:0/2347883587 >> 172.21.15.38:0/2347883587 conn(0x7fa2640f74e0 msgr2=0x7fa2640f9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:35.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.119+0000 7fa26a910700 1 -- 172.21.15.38:0/2347883587 shutdown_connections 2024-04-22T21:09:35.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.119+0000 7fa26a910700 1 -- 172.21.15.38:0/2347883587 wait complete. 2024-04-22T21:09:35.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.120+0000 7fa26a910700 1 Processor -- start 2024-04-22T21:09:35.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.120+0000 7fa26a910700 1 -- start start 2024-04-22T21:09:35.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.120+0000 7fa26a910700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa264066b80 0x7fa26410da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:35.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.120+0000 7fa26a910700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2640fffb0 0x7fa26410df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:35.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.120+0000 7fa26a910700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26410e590 con 0x7fa2640fffb0 2024-04-22T21:09:35.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.120+0000 7fa26a910700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26410ed90 con 0x7fa264066b80 2024-04-22T21:09:35.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.120+0000 7fa25bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2640fffb0 0x7fa26410df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:35.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.120+0000 7fa263fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa264066b80 0x7fa26410da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:35.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.120+0000 7fa25bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2640fffb0 0x7fa26410df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58348/0 (socket says 172.21.15.38:58348) 2024-04-22T21:09:35.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.120+0000 7fa25bfff700 1 -- 172.21.15.38:0/461195836 learned_addr learned my addr 172.21.15.38:0/461195836 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:09:35.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.120+0000 7fa25bfff700 1 -- 172.21.15.38:0/461195836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa264066b80 msgr2=0x7fa26410da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:35.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.120+0000 7fa25bfff700 1 --2- 172.21.15.38:0/461195836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa264066b80 0x7fa26410da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.120+0000 7fa25bfff700 1 -- 172.21.15.38:0/461195836 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa254005040 con 0x7fa2640fffb0 2024-04-22T21:09:35.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.121+0000 7fa25bfff700 1 --2- 172.21.15.38:0/461195836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2640fffb0 0x7fa26410df90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00eb40 tx=0x7fa24c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:35.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.121+0000 7fa2617fa700 1 -- 172.21.15.38:0/461195836 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa24c007cc0 con 0x7fa2640fffb0 2024-04-22T21:09:35.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.121+0000 7fa2617fa700 1 -- 172.21.15.38:0/461195836 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa24c00f040 con 0x7fa2640fffb0 2024-04-22T21:09:35.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.121+0000 7fa26a910700 1 -- 172.21.15.38:0/461195836 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26410f010 con 0x7fa2640fffb0 2024-04-22T21:09:35.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.121+0000 7fa26a910700 1 -- 172.21.15.38:0/461195836 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26410f550 con 0x7fa2640fffb0 2024-04-22T21:09:35.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.121+0000 7fa2617fa700 1 -- 172.21.15.38:0/461195836 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa24c00ca10 con 0x7fa2640fffb0 2024-04-22T21:09:35.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.122+0000 7fa26a910700 1 -- 172.21.15.38:0/461195836 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26405f5f0 con 0x7fa2640fffb0 2024-04-22T21:09:35.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.123+0000 7fa2617fa700 1 -- 172.21.15.38:0/461195836 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fa24c00cc80 con 0x7fa2640fffb0 2024-04-22T21:09:35.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.123+0000 7fa2617fa700 1 --2- 172.21.15.38:0/461195836 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa2500692a0 0x7fa25006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:35.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.123+0000 7fa2617fa700 1 -- 172.21.15.38:0/461195836 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa24c087000 con 0x7fa2640fffb0 2024-04-22T21:09:35.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.123+0000 7fa263fff700 1 --2- 172.21.15.38:0/461195836 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa2500692a0 0x7fa25006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:35.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.124+0000 7fa263fff700 1 --2- 172.21.15.38:0/461195836 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa2500692a0 0x7fa25006b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa254004890 tx=0x7fa254009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:35.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.128+0000 7fa2617fa700 1 -- 172.21.15.38:0/461195836 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa24c056e20 con 0x7fa2640fffb0 2024-04-22T21:09:35.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.349+0000 7fa26a910700 1 -- 172.21.15.38:0/461195836 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa26410f830 con 0x7fa2500692a0 2024-04-22T21:09:35.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.350+0000 7fa2617fa700 1 -- 172.21.15.38:0/461195836 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fa26410f830 con 0x7fa2500692a0 2024-04-22T21:09:35.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.352+0000 7fa26a910700 1 -- 172.21.15.38:0/461195836 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa2500692a0 msgr2=0x7fa25006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:35.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.352+0000 7fa26a910700 1 --2- 172.21.15.38:0/461195836 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa2500692a0 0x7fa25006b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa254004890 tx=0x7fa254009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:35.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.352+0000 7fa26a910700 1 -- 172.21.15.38:0/461195836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2640fffb0 msgr2=0x7fa26410df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:35.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.352+0000 7fa26a910700 1 --2- 172.21.15.38:0/461195836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2640fffb0 0x7fa26410df90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00eb40 tx=0x7fa24c00b040 comp rx=0 tx=0).stop 2024-04-22T21:09:35.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.352+0000 7fa26a910700 1 -- 172.21.15.38:0/461195836 shutdown_connections 2024-04-22T21:09:35.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.352+0000 7fa26a910700 1 --2- 172.21.15.38:0/461195836 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa2500692a0 0x7fa25006b750 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.352+0000 7fa26a910700 1 --2- 172.21.15.38:0/461195836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa264066b80 0x7fa26410da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.352+0000 7fa26a910700 1 --2- 172.21.15.38:0/461195836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2640fffb0 0x7fa26410df90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.352+0000 7fa26a910700 1 -- 172.21.15.38:0/461195836 >> 172.21.15.38:0/461195836 conn(0x7fa2640f74e0 msgr2=0x7fa2640f7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:35.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.353+0000 7fa26a910700 1 -- 172.21.15.38:0/461195836 shutdown_connections 2024-04-22T21:09:35.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.353+0000 7fa26a910700 1 -- 172.21.15.38:0/461195836 wait complete. 2024-04-22T21:09:35.373 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:35 smithi038 ceph-mon[26374]: pgmap v243: 65 pgs: 65 active+clean; 56 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.9 MiB/s wr, 274 op/s 2024-04-22T21:09:35.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.458+0000 7f26aab9e700 1 -- 172.21.15.38:0/3300934921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a40fcdc0 msgr2=0x7f26a40fd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:35.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.458+0000 7f26aab9e700 1 --2- 172.21.15.38:0/3300934921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a40fcdc0 0x7f26a40fd230 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2694004940 tx=0x7f2694009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:35.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.458+0000 7f26aab9e700 1 -- 172.21.15.38:0/3300934921 shutdown_connections 2024-04-22T21:09:35.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.458+0000 7f26aab9e700 1 --2- 172.21.15.38:0/3300934921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a40fcdc0 0x7f26a40fd230 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.458+0000 7f26aab9e700 1 --2- 172.21.15.38:0/3300934921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a40fbb10 0x7f26a40fbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.458+0000 7f26aab9e700 1 -- 172.21.15.38:0/3300934921 >> 172.21.15.38:0/3300934921 conn(0x7f26a40f7270 msgr2=0x7f26a40f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:35.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.458+0000 7f26aab9e700 1 -- 172.21.15.38:0/3300934921 shutdown_connections 2024-04-22T21:09:35.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.458+0000 7f26aab9e700 1 -- 172.21.15.38:0/3300934921 wait complete. 2024-04-22T21:09:35.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.459+0000 7f26aab9e700 1 Processor -- start 2024-04-22T21:09:35.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.459+0000 7f26aab9e700 1 -- start start 2024-04-22T21:09:35.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.459+0000 7f26aab9e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a40fbb10 0x7f26a410c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:35.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.459+0000 7f26aab9e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a40fcdc0 0x7f26a410cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:35.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.459+0000 7f26aab9e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a410d240 con 0x7f26a40fcdc0 2024-04-22T21:09:35.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.459+0000 7f26aab9e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a410da90 con 0x7f26a40fbb10 2024-04-22T21:09:35.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.459+0000 7f26a939b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a40fcdc0 0x7f26a410cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:35.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.459+0000 7f26a9b9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a40fbb10 0x7f26a410c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:35.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.459+0000 7f26a939b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a40fcdc0 0x7f26a410cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58364/0 (socket says 172.21.15.38:58364) 2024-04-22T21:09:35.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.459+0000 7f26a939b700 1 -- 172.21.15.38:0/2706313839 learned_addr learned my addr 172.21.15.38:0/2706313839 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:09:35.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.459+0000 7f26a939b700 1 -- 172.21.15.38:0/2706313839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a40fbb10 msgr2=0x7f26a410c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:35.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.459+0000 7f26a939b700 1 --2- 172.21.15.38:0/2706313839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a40fbb10 0x7f26a410c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.459+0000 7f26a939b700 1 -- 172.21.15.38:0/2706313839 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2694005040 con 0x7f26a40fcdc0 2024-04-22T21:09:35.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.459+0000 7f26a9b9c700 1 --2- 172.21.15.38:0/2706313839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a40fbb10 0x7f26a410c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T21:09:35.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.460+0000 7f26a939b700 1 --2- 172.21.15.38:0/2706313839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a40fcdc0 0x7f26a410cc40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2694013870 tx=0x7f2694009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:35.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.460+0000 7f269a7fc700 1 -- 172.21.15.38:0/2706313839 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2694021070 con 0x7f26a40fcdc0 2024-04-22T21:09:35.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.460+0000 7f269a7fc700 1 -- 172.21.15.38:0/2706313839 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2694009430 con 0x7f26a40fcdc0 2024-04-22T21:09:35.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.460+0000 7f26aab9e700 1 -- 172.21.15.38:0/2706313839 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a410dd10 con 0x7f26a40fcdc0 2024-04-22T21:09:35.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.460+0000 7f269a7fc700 1 -- 172.21.15.38:0/2706313839 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f269400e070 con 0x7f26a40fcdc0 2024-04-22T21:09:35.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.460+0000 7f26aab9e700 1 -- 172.21.15.38:0/2706313839 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a410e250 con 0x7f26a40fcdc0 2024-04-22T21:09:35.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.461+0000 7f26aab9e700 1 -- 172.21.15.38:0/2706313839 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a405f5f0 con 0x7f26a40fcdc0 2024-04-22T21:09:35.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.461+0000 7f269a7fc700 1 -- 172.21.15.38:0/2706313839 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f269401c750 con 0x7f26a40fcdc0 2024-04-22T21:09:35.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.462+0000 7f269a7fc700 1 --2- 172.21.15.38:0/2706313839 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f2690069230 0x7f269006b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:35.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.462+0000 7f269a7fc700 1 -- 172.21.15.38:0/2706313839 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2694088550 con 0x7f26a40fcdc0 2024-04-22T21:09:35.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.462+0000 7f26a9b9c700 1 --2- 172.21.15.38:0/2706313839 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f2690069230 0x7f269006b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:35.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.462+0000 7f26a9b9c700 1 --2- 172.21.15.38:0/2706313839 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f2690069230 0x7f269006b6e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f26a0009b80 tx=0x7f26a000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:35.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.466+0000 7f269a7fc700 1 -- 172.21.15.38:0/2706313839 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2694058220 con 0x7f26a40fcdc0 2024-04-22T21:09:35.498 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:35 smithi098 ceph-mon[34053]: pgmap v243: 65 pgs: 65 active+clean; 56 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.9 MiB/s wr, 274 op/s 2024-04-22T21:09:35.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.678+0000 7f26aab9e700 1 -- 172.21.15.38:0/2706313839 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f26a410e560 con 0x7f2690069230 2024-04-22T21:09:35.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.688+0000 7f269a7fc700 1 -- 172.21.15.38:0/2706313839 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f26a410e560 con 0x7f2690069230 2024-04-22T21:09:35.683 INFO:teuthology.orchestra.run.smithi038.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T21:09:35.683 INFO:teuthology.orchestra.run.smithi038.stdout:alertmanager.smithi038 smithi038 *:9093,9094 running (6m) 102s ago 7m 24.5M - 0.25.0 c8568f914cd2 28f5f3f389c1 2024-04-22T21:09:35.683 INFO:teuthology.orchestra.run.smithi038.stdout:crash.smithi038 smithi038 running (7m) 102s ago 7m 6665k - 17.2.7-884-g70d0cb4b cafd8ddac8fa ff0889f35e01 2024-04-22T21:09:35.683 INFO:teuthology.orchestra.run.smithi038.stdout:crash.smithi098 smithi098 running (6m) 108s ago 6m 6656k - 17.2.7-884-g70d0cb4b cafd8ddac8fa ced65820ce17 2024-04-22T21:09:35.683 INFO:teuthology.orchestra.run.smithi038.stdout:grafana.smithi038 smithi038 *:3000 running (6m) 102s ago 7m 83.2M - 9.4.7 954c08fa6188 849b80d16433 2024-04-22T21:09:35.683 INFO:teuthology.orchestra.run.smithi038.stdout:mds.cephfs.smithi038.hfegws smithi038 running (2m) 102s ago 2m 14.2M - 17.2.7-884-g70d0cb4b cafd8ddac8fa c9bc3b2faa16 2024-04-22T21:09:35.684 INFO:teuthology.orchestra.run.smithi038.stdout:mds.cephfs.smithi038.vmivbd smithi038 running (2m) 102s ago 2m 10.4M - 17.2.7-884-g70d0cb4b cafd8ddac8fa 5cc4c9d5a63c 2024-04-22T21:09:35.684 INFO:teuthology.orchestra.run.smithi038.stdout:mds.cephfs.smithi098.izcoza smithi098 running (118s) 108s ago 117s 14.6M - 17.2.7-884-g70d0cb4b cafd8ddac8fa 3ef691591a41 2024-04-22T21:09:35.684 INFO:teuthology.orchestra.run.smithi038.stdout:mds.cephfs.smithi098.mipeta smithi098 running (2m) 108s ago 2m 14.2M - 17.2.7-884-g70d0cb4b cafd8ddac8fa 539456953f6c 2024-04-22T21:09:35.684 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.smithi038.lxfgut smithi038 *:9283 running (8m) 102s ago 8m 481M - 17.2.7-884-g70d0cb4b cafd8ddac8fa ee36b4dede58 2024-04-22T21:09:35.684 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.smithi098.azqohf smithi098 *:8443,9283 running (6m) 108s ago 6m 435M - 17.2.7-884-g70d0cb4b cafd8ddac8fa c3fae1c785a2 2024-04-22T21:09:35.684 INFO:teuthology.orchestra.run.smithi038.stdout:mon.smithi038 smithi038 running (8m) 102s ago 9m 45.3M 2048M 17.2.7-884-g70d0cb4b cafd8ddac8fa 2137696f8742 2024-04-22T21:09:35.684 INFO:teuthology.orchestra.run.smithi038.stdout:mon.smithi098 smithi098 running (6m) 108s ago 6m 35.3M 2048M 17.2.7-884-g70d0cb4b cafd8ddac8fa e996b8c559ed 2024-04-22T21:09:35.684 INFO:teuthology.orchestra.run.smithi038.stdout:node-exporter.smithi038 smithi038 *:9100 running (7m) 102s ago 7m 15.7M - 1.5.0 0da6a335fe13 347a9e37455f 2024-04-22T21:09:35.685 INFO:teuthology.orchestra.run.smithi038.stdout:node-exporter.smithi098 smithi098 *:9100 running (6m) 108s ago 6m 17.2M - 1.5.0 0da6a335fe13 5274c36423cc 2024-04-22T21:09:35.685 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 smithi038 running (5m) 102s ago 5m 46.6M 4289M 17.2.7-884-g70d0cb4b cafd8ddac8fa 160468eff0dc 2024-04-22T21:09:35.685 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 smithi038 running (5m) 102s ago 5m 47.9M 4289M 17.2.7-884-g70d0cb4b cafd8ddac8fa d79042851a05 2024-04-22T21:09:35.685 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2 smithi038 running (4m) 102s ago 4m 46.7M 4289M 17.2.7-884-g70d0cb4b cafd8ddac8fa 8aaa1e69183c 2024-04-22T21:09:35.685 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3 smithi098 running (4m) 108s ago 4m 45.2M 5312M 17.2.7-884-g70d0cb4b cafd8ddac8fa a59b0f44b178 2024-04-22T21:09:35.685 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 smithi098 running (3m) 108s ago 3m 41.2M 5312M 17.2.7-884-g70d0cb4b cafd8ddac8fa 848d77af5440 2024-04-22T21:09:35.685 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 smithi098 running (3m) 108s ago 3m 43.6M 5312M 17.2.7-884-g70d0cb4b cafd8ddac8fa ef4d55a626c6 2024-04-22T21:09:35.685 INFO:teuthology.orchestra.run.smithi038.stdout:prometheus.smithi038 smithi038 *:9095 running (5m) 102s ago 7m 55.0M - 2.43.0 a07b618ecd1d 5cc3d358fbbd 2024-04-22T21:09:35.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.690+0000 7f26aab9e700 1 -- 172.21.15.38:0/2706313839 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f2690069230 msgr2=0x7f269006b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:35.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.690+0000 7f26aab9e700 1 --2- 172.21.15.38:0/2706313839 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f2690069230 0x7f269006b6e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f26a0009b80 tx=0x7f26a000b040 comp rx=0 tx=0).stop 2024-04-22T21:09:35.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.690+0000 7f26aab9e700 1 -- 172.21.15.38:0/2706313839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a40fcdc0 msgr2=0x7f26a410cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:35.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.690+0000 7f26aab9e700 1 --2- 172.21.15.38:0/2706313839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a40fcdc0 0x7f26a410cc40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2694013870 tx=0x7f2694009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:35.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.690+0000 7f26aab9e700 1 -- 172.21.15.38:0/2706313839 shutdown_connections 2024-04-22T21:09:35.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.690+0000 7f26aab9e700 1 --2- 172.21.15.38:0/2706313839 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f2690069230 0x7f269006b6e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.690+0000 7f26aab9e700 1 --2- 172.21.15.38:0/2706313839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a40fbb10 0x7f26a410c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.690+0000 7f26aab9e700 1 --2- 172.21.15.38:0/2706313839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a40fcdc0 0x7f26a410cc40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.690+0000 7f26aab9e700 1 -- 172.21.15.38:0/2706313839 >> 172.21.15.38:0/2706313839 conn(0x7f26a40f7270 msgr2=0x7f26a40f8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:35.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.690+0000 7f26aab9e700 1 -- 172.21.15.38:0/2706313839 shutdown_connections 2024-04-22T21:09:35.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.690+0000 7f26aab9e700 1 -- 172.21.15.38:0/2706313839 wait complete. 2024-04-22T21:09:35.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.798+0000 7fa31b1d4700 1 -- 172.21.15.38:0/1950892599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3140fffe0 msgr2=0x7fa3141023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:35.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.798+0000 7fa31b1d4700 1 --2- 172.21.15.38:0/1950892599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3140fffe0 0x7fa3141023c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa300004940 tx=0x7fa300009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:35.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.798+0000 7fa31b1d4700 1 -- 172.21.15.38:0/1950892599 shutdown_connections 2024-04-22T21:09:35.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.798+0000 7fa31b1d4700 1 --2- 172.21.15.38:0/1950892599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3140fffe0 0x7fa3141023c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.798+0000 7fa31b1d4700 1 --2- 172.21.15.38:0/1950892599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa314066ea0 0x7fa3140ffaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.798+0000 7fa31b1d4700 1 -- 172.21.15.38:0/1950892599 >> 172.21.15.38:0/1950892599 conn(0x7fa3140f7450 msgr2=0x7fa3140f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:35.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.798+0000 7fa31b1d4700 1 -- 172.21.15.38:0/1950892599 shutdown_connections 2024-04-22T21:09:35.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.798+0000 7fa31b1d4700 1 -- 172.21.15.38:0/1950892599 wait complete. 2024-04-22T21:09:35.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.799+0000 7fa31b1d4700 1 Processor -- start 2024-04-22T21:09:35.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.799+0000 7fa31b1d4700 1 -- start start 2024-04-22T21:09:35.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.799+0000 7fa31b1d4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa314066ea0 0x7fa31410dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:35.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.799+0000 7fa31b1d4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3140fffe0 0x7fa31410e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:35.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.799+0000 7fa31b1d4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31410e760 con 0x7fa314066ea0 2024-04-22T21:09:35.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.799+0000 7fa31b1d4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31410e8d0 con 0x7fa3140fffe0 2024-04-22T21:09:35.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.799+0000 7fa318f70700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa314066ea0 0x7fa31410dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:35.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.799+0000 7fa318f70700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa314066ea0 0x7fa31410dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58386/0 (socket says 172.21.15.38:58386) 2024-04-22T21:09:35.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.799+0000 7fa318f70700 1 -- 172.21.15.38:0/2843461867 learned_addr learned my addr 172.21.15.38:0/2843461867 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:09:35.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.799+0000 7fa313fff700 1 --2- 172.21.15.38:0/2843461867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3140fffe0 0x7fa31410e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:35.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.800+0000 7fa318f70700 1 -- 172.21.15.38:0/2843461867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3140fffe0 msgr2=0x7fa31410e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:35.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.800+0000 7fa318f70700 1 --2- 172.21.15.38:0/2843461867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3140fffe0 0x7fa31410e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:35.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.800+0000 7fa318f70700 1 -- 172.21.15.38:0/2843461867 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa300005040 con 0x7fa314066ea0 2024-04-22T21:09:35.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.800+0000 7fa313fff700 1 --2- 172.21.15.38:0/2843461867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3140fffe0 0x7fa31410e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T21:09:35.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.800+0000 7fa318f70700 1 --2- 172.21.15.38:0/2843461867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa314066ea0 0x7fa31410dc20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa30800eb70 tx=0x7fa30800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:35.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.800+0000 7fa3117fa700 1 -- 172.21.15.38:0/2843461867 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa308007cc0 con 0x7fa314066ea0 2024-04-22T21:09:35.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.800+0000 7fa3117fa700 1 -- 172.21.15.38:0/2843461867 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa30800f040 con 0x7fa314066ea0 2024-04-22T21:09:35.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.800+0000 7fa31b1d4700 1 -- 172.21.15.38:0/2843461867 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3141132d0 con 0x7fa314066ea0 2024-04-22T21:09:35.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.800+0000 7fa3117fa700 1 -- 172.21.15.38:0/2843461867 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa30800cb20 con 0x7fa314066ea0 2024-04-22T21:09:35.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.800+0000 7fa31b1d4700 1 -- 172.21.15.38:0/2843461867 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa314113810 con 0x7fa314066ea0 2024-04-22T21:09:35.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.801+0000 7fa31b1d4700 1 -- 172.21.15.38:0/2843461867 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31404c710 con 0x7fa314066ea0 2024-04-22T21:09:35.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.802+0000 7fa3117fa700 1 -- 172.21.15.38:0/2843461867 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fa30800cc80 con 0x7fa314066ea0 2024-04-22T21:09:35.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.802+0000 7fa3117fa700 1 --2- 172.21.15.38:0/2843461867 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa3040692a0 0x7fa30406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:35.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.802+0000 7fa3117fa700 1 -- 172.21.15.38:0/2843461867 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa308086c20 con 0x7fa314066ea0 2024-04-22T21:09:35.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.802+0000 7fa313fff700 1 --2- 172.21.15.38:0/2843461867 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa3040692a0 0x7fa30406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:35.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.802+0000 7fa313fff700 1 --2- 172.21.15.38:0/2843461867 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa3040692a0 0x7fa30406b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa300004890 tx=0x7fa300009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:35.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:35.806+0000 7fa3117fa700 1 -- 172.21.15.38:0/2843461867 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa30804e250 con 0x7fa314066ea0 2024-04-22T21:09:36.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.111+0000 7fa31b1d4700 1 -- 172.21.15.38:0/2843461867 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa314113af0 con 0x7fa314066ea0 2024-04-22T21:09:36.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.112+0000 7fa3117fa700 1 -- 172.21.15.38:0/2843461867 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fa3080569c0 con 0x7fa314066ea0 2024-04-22T21:09:36.107 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T21:09:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: "mon": { 2024-04-22T21:09:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph version 17.2.7-884-g70d0cb4b (70d0cb4b9d576828afb0a05dcf8732894d343ece) quincy (stable)": 2 2024-04-22T21:09:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T21:09:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": { 2024-04-22T21:09:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph version 17.2.7-884-g70d0cb4b (70d0cb4b9d576828afb0a05dcf8732894d343ece) quincy (stable)": 2 2024-04-22T21:09:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T21:09:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": { 2024-04-22T21:09:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph version 17.2.7-884-g70d0cb4b (70d0cb4b9d576828afb0a05dcf8732894d343ece) quincy (stable)": 6 2024-04-22T21:09:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T21:09:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: "mds": { 2024-04-22T21:09:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph version 17.2.7-884-g70d0cb4b (70d0cb4b9d576828afb0a05dcf8732894d343ece) quincy (stable)": 4 2024-04-22T21:09:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T21:09:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: "overall": { 2024-04-22T21:09:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph version 17.2.7-884-g70d0cb4b (70d0cb4b9d576828afb0a05dcf8732894d343ece) quincy (stable)": 14 2024-04-22T21:09:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T21:09:36.108 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T21:09:36.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.114+0000 7fa31b1d4700 1 -- 172.21.15.38:0/2843461867 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa3040692a0 msgr2=0x7fa30406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:36.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.114+0000 7fa31b1d4700 1 --2- 172.21.15.38:0/2843461867 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa3040692a0 0x7fa30406b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa300004890 tx=0x7fa300009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:36.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.114+0000 7fa31b1d4700 1 -- 172.21.15.38:0/2843461867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa314066ea0 msgr2=0x7fa31410dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:36.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.114+0000 7fa31b1d4700 1 --2- 172.21.15.38:0/2843461867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa314066ea0 0x7fa31410dc20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa30800eb70 tx=0x7fa30800b040 comp rx=0 tx=0).stop 2024-04-22T21:09:36.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.114+0000 7fa31b1d4700 1 -- 172.21.15.38:0/2843461867 shutdown_connections 2024-04-22T21:09:36.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.114+0000 7fa31b1d4700 1 --2- 172.21.15.38:0/2843461867 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7fa3040692a0 0x7fa30406b750 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:36.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.114+0000 7fa31b1d4700 1 --2- 172.21.15.38:0/2843461867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa314066ea0 0x7fa31410dc20 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:36.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.114+0000 7fa31b1d4700 1 --2- 172.21.15.38:0/2843461867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3140fffe0 0x7fa31410e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:36.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.114+0000 7fa31b1d4700 1 -- 172.21.15.38:0/2843461867 >> 172.21.15.38:0/2843461867 conn(0x7fa3140f7450 msgr2=0x7fa3140f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:36.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.114+0000 7fa31b1d4700 1 -- 172.21.15.38:0/2843461867 shutdown_connections 2024-04-22T21:09:36.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.114+0000 7fa31b1d4700 1 -- 172.21.15.38:0/2843461867 wait complete. 2024-04-22T21:09:36.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.219+0000 7f2b624c3700 1 -- 172.21.15.38:0/2417911111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b5c0fcff0 msgr2=0x7f2b5c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:36.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.219+0000 7f2b624c3700 1 --2- 172.21.15.38:0/2417911111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b5c0fcff0 0x7f2b5c0fd460 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2b50004940 tx=0x7f2b50009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:36.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.219+0000 7f2b624c3700 1 -- 172.21.15.38:0/2417911111 shutdown_connections 2024-04-22T21:09:36.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.219+0000 7f2b624c3700 1 --2- 172.21.15.38:0/2417911111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b5c0fcff0 0x7f2b5c0fd460 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:36.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.219+0000 7f2b624c3700 1 --2- 172.21.15.38:0/2417911111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b5c0fbd40 0x7f2b5c0fc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:36.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.219+0000 7f2b624c3700 1 -- 172.21.15.38:0/2417911111 >> 172.21.15.38:0/2417911111 conn(0x7f2b5c0f74c0 msgr2=0x7f2b5c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:36.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.219+0000 7f2b624c3700 1 -- 172.21.15.38:0/2417911111 shutdown_connections 2024-04-22T21:09:36.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.219+0000 7f2b624c3700 1 -- 172.21.15.38:0/2417911111 wait complete. 2024-04-22T21:09:36.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.220+0000 7f2b624c3700 1 Processor -- start 2024-04-22T21:09:36.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.220+0000 7f2b624c3700 1 -- start start 2024-04-22T21:09:36.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.220+0000 7f2b624c3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b5c0fbd40 0x7f2b5c10c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:36.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.220+0000 7f2b624c3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b5c0fcff0 0x7f2b5c10ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:36.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.220+0000 7f2b624c3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c10d400 con 0x7f2b5c0fbd40 2024-04-22T21:09:36.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.220+0000 7f2b624c3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c10dc50 con 0x7f2b5c0fcff0 2024-04-22T21:09:36.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.220+0000 7f2b5bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b5c0fbd40 0x7f2b5c10c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:36.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.220+0000 7f2b5bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b5c0fbd40 0x7f2b5c10c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58394/0 (socket says 172.21.15.38:58394) 2024-04-22T21:09:36.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.220+0000 7f2b5bfff700 1 -- 172.21.15.38:0/3290962475 learned_addr learned my addr 172.21.15.38:0/3290962475 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T21:09:36.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.220+0000 7f2b5b7fe700 1 --2- 172.21.15.38:0/3290962475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b5c0fcff0 0x7f2b5c10ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:36.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.220+0000 7f2b5bfff700 1 -- 172.21.15.38:0/3290962475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b5c0fcff0 msgr2=0x7f2b5c10ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:36.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.220+0000 7f2b5bfff700 1 --2- 172.21.15.38:0/3290962475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b5c0fcff0 0x7f2b5c10ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:36.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.220+0000 7f2b5bfff700 1 -- 172.21.15.38:0/3290962475 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b50005040 con 0x7f2b5c0fbd40 2024-04-22T21:09:36.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.221+0000 7f2b5bfff700 1 --2- 172.21.15.38:0/3290962475 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b5c0fbd40 0x7f2b5c10c8c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c010d80 tx=0x7f2b4c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:36.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.221+0000 7f2b58ff9700 1 -- 172.21.15.38:0/3290962475 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b4c010e80 con 0x7f2b5c0fbd40 2024-04-22T21:09:36.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.221+0000 7f2b624c3700 1 -- 172.21.15.38:0/3290962475 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5c10ded0 con 0x7f2b5c0fbd40 2024-04-22T21:09:36.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.221+0000 7f2b58ff9700 1 -- 172.21.15.38:0/3290962475 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b4c0090d0 con 0x7f2b5c0fbd40 2024-04-22T21:09:36.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.221+0000 7f2b58ff9700 1 -- 172.21.15.38:0/3290962475 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b4c012670 con 0x7f2b5c0fbd40 2024-04-22T21:09:36.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.221+0000 7f2b624c3700 1 -- 172.21.15.38:0/3290962475 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5c10e410 con 0x7f2b5c0fbd40 2024-04-22T21:09:36.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.222+0000 7f2b624c3700 1 -- 172.21.15.38:0/3290962475 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5c05f5f0 con 0x7f2b5c0fbd40 2024-04-22T21:09:36.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.222+0000 7f2b58ff9700 1 -- 172.21.15.38:0/3290962475 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f2b4c018030 con 0x7f2b5c0fbd40 2024-04-22T21:09:36.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.223+0000 7f2b58ff9700 1 --2- 172.21.15.38:0/3290962475 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f2b440692a0 0x7f2b4406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T21:09:36.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.223+0000 7f2b58ff9700 1 -- 172.21.15.38:0/3290962475 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2b4c016070 con 0x7f2b5c0fbd40 2024-04-22T21:09:36.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.223+0000 7f2b5b7fe700 1 --2- 172.21.15.38:0/3290962475 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f2b440692a0 0x7f2b4406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T21:09:36.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.223+0000 7f2b5b7fe700 1 --2- 172.21.15.38:0/3290962475 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f2b440692a0 0x7f2b4406b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2b50009b30 tx=0x7f2b50009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T21:09:36.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.227+0000 7f2b58ff9700 1 -- 172.21.15.38:0/3290962475 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2b4c059410 con 0x7f2b5c0fbd40 2024-04-22T21:09:36.373 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:36 smithi038 ceph-mon[26374]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:09:36.373 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:36 smithi038 ceph-mon[26374]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:09:36.373 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:36 smithi038 ceph-mon[26374]: from='client.? 172.21.15.38:0/2843461867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:09:36.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.440+0000 7f2b624c3700 1 -- 172.21.15.38:0/3290962475 --> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b5c10e720 con 0x7f2b440692a0 2024-04-22T21:09:36.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.441+0000 7f2b58ff9700 1 -- 172.21.15.38:0/3290962475 <== mgr.14212 v2:172.21.15.38:6800/368620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f2b5c10e720 con 0x7f2b440692a0 2024-04-22T21:09:36.436 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T21:09:36.436 INFO:teuthology.orchestra.run.smithi038.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:90c23f6ff09279e806b61a7a0a9b6863a8d239088a6c790da6364c3d6ab9ae8d", 2024-04-22T21:09:36.436 INFO:teuthology.orchestra.run.smithi038.stdout: "in_progress": true, 2024-04-22T21:09:36.437 INFO:teuthology.orchestra.run.smithi038.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T21:09:36.437 INFO:teuthology.orchestra.run.smithi038.stdout: "services_complete": [], 2024-04-22T21:09:36.437 INFO:teuthology.orchestra.run.smithi038.stdout: "progress": "0/2 daemons upgraded", 2024-04-22T21:09:36.437 INFO:teuthology.orchestra.run.smithi038.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:90c23f6ff09279e806b61a7a0a9b6863a8d239088a6c790da6364c3d6ab9ae8d image on host smithi098", 2024-04-22T21:09:36.437 INFO:teuthology.orchestra.run.smithi038.stdout: "is_paused": false 2024-04-22T21:09:36.437 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T21:09:36.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.443+0000 7f2b624c3700 1 -- 172.21.15.38:0/3290962475 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f2b440692a0 msgr2=0x7f2b4406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:36.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.443+0000 7f2b624c3700 1 --2- 172.21.15.38:0/3290962475 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f2b440692a0 0x7f2b4406b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2b50009b30 tx=0x7f2b50009e30 comp rx=0 tx=0).stop 2024-04-22T21:09:36.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.443+0000 7f2b624c3700 1 -- 172.21.15.38:0/3290962475 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b5c0fbd40 msgr2=0x7f2b5c10c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T21:09:36.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.443+0000 7f2b624c3700 1 --2- 172.21.15.38:0/3290962475 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b5c0fbd40 0x7f2b5c10c8c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c010d80 tx=0x7f2b4c00b040 comp rx=0 tx=0).stop 2024-04-22T21:09:36.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.443+0000 7f2b624c3700 1 -- 172.21.15.38:0/3290962475 shutdown_connections 2024-04-22T21:09:36.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.443+0000 7f2b624c3700 1 --2- 172.21.15.38:0/3290962475 >> [v2:172.21.15.38:6800/368620,v1:172.21.15.38:6801/368620] conn(0x7f2b440692a0 0x7f2b4406b750 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:36.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.443+0000 7f2b624c3700 1 --2- 172.21.15.38:0/3290962475 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b5c0fbd40 0x7f2b5c10c8c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:36.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.443+0000 7f2b624c3700 1 --2- 172.21.15.38:0/3290962475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b5c0fcff0 0x7f2b5c10ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T21:09:36.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.443+0000 7f2b624c3700 1 -- 172.21.15.38:0/3290962475 >> 172.21.15.38:0/3290962475 conn(0x7f2b5c0f74c0 msgr2=0x7f2b5c0f8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T21:09:36.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.443+0000 7f2b624c3700 1 -- 172.21.15.38:0/3290962475 shutdown_connections 2024-04-22T21:09:36.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T21:09:36.443+0000 7f2b624c3700 1 -- 172.21.15.38:0/3290962475 wait complete. 2024-04-22T21:09:36.498 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:36 smithi098 ceph-mon[34053]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:09:36.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:36 smithi098 ceph-mon[34053]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:09:36.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:36 smithi098 ceph-mon[34053]: from='client.? 172.21.15.38:0/2843461867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:09:37.498 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:37 smithi098 ceph-mon[34053]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:09:37.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:37 smithi098 ceph-mon[34053]: pgmap v244: 65 pgs: 65 active+clean; 72 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.0 MiB/s wr, 424 op/s 2024-04-22T21:09:37.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:37 smithi038 ceph-mon[26374]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:09:37.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:37 smithi038 ceph-mon[26374]: pgmap v244: 65 pgs: 65 active+clean; 72 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.0 MiB/s wr, 424 op/s 2024-04-22T21:09:38.498 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:38 smithi098 ceph-mon[34053]: from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:09:38.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:38 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:09:38.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:38 smithi038 ceph-mon[26374]: from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:09:38.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:38 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:09:39.498 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:39 smithi098 ceph-mon[34053]: pgmap v245: 65 pgs: 65 active+clean; 89 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.3 MiB/s wr, 533 op/s 2024-04-22T21:09:39.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:39 smithi038 ceph-mon[26374]: pgmap v245: 65 pgs: 65 active+clean; 89 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.3 MiB/s wr, 533 op/s 2024-04-22T21:09:41.498 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:41 smithi098 ceph-mon[34053]: pgmap v246: 65 pgs: 65 active+clean; 97 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.3 MiB/s wr, 373 op/s 2024-04-22T21:09:41.537 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:41 smithi038 ceph-mon[26374]: pgmap v246: 65 pgs: 65 active+clean; 97 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.3 MiB/s wr, 373 op/s 2024-04-22T21:09:43.498 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:43 smithi098 ceph-mon[34053]: pgmap v247: 65 pgs: 65 active+clean; 117 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 5.7 MiB/s wr, 608 op/s 2024-04-22T21:09:43.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:43 smithi038 ceph-mon[26374]: pgmap v247: 65 pgs: 65 active+clean; 117 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 5.7 MiB/s wr, 608 op/s 2024-04-22T21:09:45.498 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:45 smithi098 ceph-mon[34053]: pgmap v248: 65 pgs: 65 active+clean; 122 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 5.9 MiB/s wr, 692 op/s 2024-04-22T21:09:45.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:45 smithi038 ceph-mon[26374]: pgmap v248: 65 pgs: 65 active+clean; 122 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 5.9 MiB/s wr, 692 op/s 2024-04-22T21:09:47.320 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:47 smithi098 ceph-mon[34053]: pgmap v249: 65 pgs: 65 active+clean; 139 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 7.5 MiB/s wr, 982 op/s 2024-04-22T21:09:47.320 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:47 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:09:47.320 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:47 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi098.azqohf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:09:47.320 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:47 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:09:47.320 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:47 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:09:47.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:47 smithi038 ceph-mon[26374]: pgmap v249: 65 pgs: 65 active+clean; 139 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 7.5 MiB/s wr, 982 op/s 2024-04-22T21:09:47.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:47 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:09:47.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:47 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi098.azqohf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:09:47.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:47 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:09:47.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:47 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:09:48.386 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:48 smithi098 ceph-mon[34053]: Upgrade: Updating mgr.smithi098.azqohf 2024-04-22T21:09:48.387 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:48 smithi098 ceph-mon[34053]: Deploying daemon mgr.smithi098.azqohf on smithi098 2024-04-22T21:09:48.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:48 smithi038 ceph-mon[26374]: Upgrade: Updating mgr.smithi098.azqohf 2024-04-22T21:09:48.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:48 smithi038 ceph-mon[26374]: Deploying daemon mgr.smithi098.azqohf on smithi098 2024-04-22T21:09:49.498 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:49 smithi098 ceph-mon[34053]: pgmap v250: 65 pgs: 65 active+clean; 157 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 7.7 MiB/s wr, 994 op/s 2024-04-22T21:09:49.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:49 smithi038 ceph-mon[26374]: pgmap v250: 65 pgs: 65 active+clean; 157 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 7.7 MiB/s wr, 994 op/s 2024-04-22T21:09:51.498 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:51 smithi098 ceph-mon[34053]: pgmap v251: 65 pgs: 65 active+clean; 161 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 6.5 MiB/s wr, 781 op/s 2024-04-22T21:09:51.534 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:51 smithi038 ceph-mon[26374]: pgmap v251: 65 pgs: 65 active+clean; 161 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 6.5 MiB/s wr, 781 op/s 2024-04-22T21:09:53.498 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:53 smithi098 ceph-mon[34053]: pgmap v252: 65 pgs: 65 active+clean; 172 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 6.7 MiB/s wr, 797 op/s 2024-04-22T21:09:53.498 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:53 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:09:53.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:53 smithi038 ceph-mon[26374]: pgmap v252: 65 pgs: 65 active+clean; 172 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 6.7 MiB/s wr, 797 op/s 2024-04-22T21:09:53.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:53 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T21:09:53.905 INFO:tasks.workunit.client.1.smithi098.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-22T21:09:53.918 INFO:tasks.workunit.client.1.smithi098.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-22T21:09:53.919 INFO:tasks.workunit.client.1.smithi098.stderr:+ make 2024-04-22T21:09:54.406 INFO:tasks.workunit.client.1.smithi098.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-22T21:09:55.061 INFO:tasks.workunit.client.0.smithi038.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-22T21:09:55.061 INFO:tasks.workunit.client.0.smithi038.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-22T21:09:55.061 INFO:tasks.workunit.client.0.smithi038.stderr:+ make 2024-04-22T21:09:55.498 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:55 smithi098 ceph-mon[34053]: pgmap v253: 65 pgs: 65 active+clean; 172 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 4.9 MiB/s wr, 553 op/s 2024-04-22T21:09:55.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:55 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:09:55.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:55 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:09:55.499 INFO:journalctl@ceph.mon.smithi098.smithi098.stdout:Apr 22 21:09:55 smithi098 ceph-mon[34053]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:09:55.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:55 smithi038 ceph-mon[26374]: pgmap v253: 65 pgs: 65 active+clean; 172 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 4.9 MiB/s wr, 553 op/s 2024-04-22T21:09:55.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:55 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:09:55.623 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:55 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' 2024-04-22T21:09:55.624 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Apr 22 21:09:55 smithi038 ceph-mon[26374]: from='mgr.14212 172.21.15.38:0/867579687' entity='mgr.smithi038.lxfgut' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:09:56.444 INFO:tasks.workunit.client.0.smithi038.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-23T08:54:33.993 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-04-23T08:54:34.023 DEBUG:teuthology.task.console_log:Killing console logger for smithi038 2024-04-23T08:54:34.023 DEBUG:teuthology.task.console_log:Killing console logger for smithi098 2024-04-23T08:54:34.023 DEBUG:teuthology.exit:Finished running handlers